When JSON Is Not Enough: Semantic Reliability of Schema-Constrained LLM Ordering Agents

AI 中文解读

JSON格式正确就是AI的“免死金牌”？错了！最新研究发现，即便AI输出的数据结构完美无缺，内容照样可能一塌糊涂。研究人员专门为餐厅点餐场景设计了OrderBench测试，让四个主流大模型在纯文本和JSON格式两种模式下点单，结果发现：最强模型虽然格式正确率达到了100%，但语义准确率却只有80%——也就是说，每五次点单就有一回点错菜，比如点了菜单上根本没有的菜、重复下单，甚至答应了不该接的单。在弱一点的模型上，这种“格式对但内容错”的错误率更是高达两位数。通俗来说，这就好比AI帮你填表格，表格格式是标准的，填的位置也对，但填的内容却是一堆毫无意义或违规的数字。以前我们担心AI输出格式混乱导致系统崩溃，现在发现格式对了不等于事情办对了。这个发现对普通人意义重大：未来你让AI助手帮你订外卖、预约服务、操作设备，它可能严格按照要求生成了一条正确的指令，但实际执行的内容却是错的——比如点了十份你过敏的菜品。简单说，光靠格式规范还不够，必须得有人工审核或兜底机制，否则AI越“听话”，闯的祸可能越大。

arXiv:2607.18261v1 Announce Type: new Abstract: LLM agents are increasingly used as transaction compilers: a user states an intent in natural language, and the model emits a structured object that an API can execute. JSON Schema and provider-level structured-output modes are useful because they remove a large class of parse failures, but they do not by themselves decide whether the object is a safe, faithful transaction. We introduce OrderBench, a deterministic benchmark for restaurant ordering agents that separates syntactic validity, schema validity, status decisions, exact item semantics, constraint preservation, and unsafe acceptances. Across 2,400 Nebius Token Factory calls to four open models in prompt-only and JSON-schema modes, we find that schema-valid output can still have large semantic error rates. In the strongest model, both modes achieve 100% schema validity, yet semantic success remains near 80%; in weaker models, schema-valid unsafe acceptances occur in double digits. The result is a concrete engineering warning: structured output is a necessary interface layer, not a substitute for domain verification and fail-closed execution.