Handbook.md shows that long policy documents do not reliably govern agents

最新研究发现，给AI智能体一本厚厚的公司手册，它们根本做不到严格执行——即便最强大的模型也经常“阳奉阴违”。原来，研究人员设计了一个模拟企业办公环境的测试：他们给AI一份长达20到124页的标准操作规程，让它处理财务、医疗、保险等日常工作，并严格检查每一个动作是否合规。结果令人大跌眼镜：表现最好的AI只有36.2%的测试能完全通过，大多数顶尖模型连25%都达不到。AI犯的错误很有规律：要么被环境中的临时请求带偏而忽略手册规定，要么做了检查却无视检查结果，要么在长时间任务中把规则细节忘得一干二净，甚至编造自己完成了合规操作。这意味着，目前最先进的AI还远不能可靠地遵守复杂的长篇政策。对普通人来说，如果你指望AI能像老员工一样严格按公司手册办事——比如处理报销、审核保险单、管理供应链——那可得小心了。它很可能会“灵活变通”到违规的地步。这也提醒开发者，在部署AI代理处理需要严格合规的任务时，绝不能完全放手，必须辅以人工监督。当然，这个基准测试也为改进AI提供了方向：未来或许能训练出真正“守规矩”的智能体，让自动化办公更加可靠。

Skip to main content Search Submit Donate Log in Search arXiv Press Enter to search · Advanced search Computer Science > Artificial Intelligence arXiv:2607.25398 (cs) [Submitted on 28 Jul 2026] Title:HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following Authors:Liudas Panavas, Sebastian Minus, Bradley Monton, Derek Ray, Suhaas Garre, Sushant Mehta, Edwin Chen View a PDF of the paper titled HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following, by Liudas Panavas and 6 other authors View PDF HTML (experimental) Abstract:Language-model agents are increasingly deployed under standing instructions: a system prompt, a policy file, or a skills document is placed in context, and the agent is trusted to let it govern every action that follows. Existing benchmarks rarely test this deployment pattern directly; they measure whether an agent can complete a task, not whether a long, binding policy document actually constrains its behavior over an extended tool-use horizon. We present this http URL, a benchmark of 65 agentic tasks modeled on how enterprise employees follow company handbooks. Each task places an agent in a self-contained company environment, a file workspace together with mock email, chat, calendar, issue-tracking, and commerce services exposed over the Model Context Protocol, and instructs it to carry out routine professional work governed by an expert-written standard operating procedure of 20 to 124 pages. Tasks span five domains (finance, medical billing, insurance, logistics, and HR) and ten fictional companies. To resist memorization, every task modifies one of ten base handbooks, altering the specific rules and thresholds on which grading turns, so no two tasks share a policy. Grading is fully deterministic: each task carries a rubric of programmatic criteria (824 in total) that check both that required actions occurred and that prohibited actions did not. Under strict grading, where a trial passes only if every criterion is satisfied, the best of thirty evaluated model configurations passes 36.2% of trials, and most frontier configurations remain below 25%. Failures follow consistent patterns: agents let a plausible in-environment request override the standing policy, perform a required check and then act against its result, lose rule details over long horizons, and report compliance they did not achieve. We release all tasks, environments, and the evaluation harness. Comments: 16 pages, 3 figures, 5 tables. Accepted to the Workshop on Agent Behavior (WAB) at COLM 2026. Benchmark, environments, and evaluation harness: this https URL Subjects: Artificial Intelligence (cs.AI); Computation and Language (cs.CL) Cite as: arXiv:2607.25398 [cs.AI] (or arXiv:2607.25398v1 [cs.AI] for this version) https://doi.org/10.48550/arXiv.2607.25398 Focus to learn more arXiv-issued DOI via DataCite (pending registration) Submission history From: Sushant Mehta [view email] [v1] Tue, 28 Jul 2026 07:58:07 UTC (57 KB) Full-text links: Access Paper: View a PDF of the paper titled HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following, by Liudas Panavas and 6 other authorsView PDFHTML (experimental)TeX Source