Daily Tech Briefing
AI 科技速览
每天 5 分钟内学习 AI。获取最新的人工智能新闻,理解其重要性,并学习如何将其应用于您的工作。
arXiv AI · 2026/8/2 05:00:44
Judging Is Not Enumerating: Silent Omissions in LLM-Authored Acceptable Sets
AI 中文解读
大模型当“出题人”翻车了!最新研究发现,AI自己编写测试题和评判标准时,会严重“漏题”——明明能准确判断答案对错,却写不全正确答案。就像老师知道这道题怎么做,但出试卷时却漏掉了关键选项。
研究人员用代码测试和词汇表考察AI,发现它们“批改”能力很强(准确率80%以上),但“出题”时只让不到四成真正正确的方案通过。更棘手的是,这种错误全是“遗漏”而非“错判”,就像购物清单漏写了牛奶,你根本不知道漏了什么。AI发现故意掺入的错误答案的概率,比发现遗漏答案高六七倍。
好消息是,把AI写的错误答案改成标准答案后,测试质量能提升3到10倍,这项技术可大幅提升AI处理复杂任务的可靠性。未来AI不仅要做“答题者”,还要当“出题者”时,这套纠错机制将是保障准确性的关键。
Language models are increasingly promoted from examinees to examiners: they write the test suites, answer keys, rubrics, and reward functions that define correctness for other systems. We measure the capability that role assumes and find it lacking under the protocol the role is usually deployed with, one-shot greedy authoring with no test-time reasoning. Across four reference constructions - two with complete finite truth, one with a hardened executable reference (HumanEval+/MBPP+), one with an explicitly incomplete lexical reference (WordNet) - models judge whether a candidate belongs far better than they author the set itself. On the incompleteness-proof algorithmic construction the gap is +0.34 to +0.29 F1 over a 24x parameter range and does not close; on executable code, models judging at F1 0.74-0.90 author suites admitting only 19-42% of oracle-correct solutions. A control locates the deficit: asked to emit the predicate rather than its extension, the same models reach F1 about 0.99. The failure is not missing knowledge or an inability to specify, but an inability to materialise the region a specification induces. The dominant error is omission, which resists audit: an over-inclusion is a token a reviewer can challenge, a missing member an absence whose discovery is the authoring problem itself. Models detect planted over-inclusions 6-7x more often than planted omissions, and a production deployment of 43,227 items fails omission-first at 10:1. Wired into RLVR, an authored key costs 1.9 points of accuracy against an exact oracle and 18.5 WordNet-relative (six paired seeds, p=0.031). Gating authored verifiers on a known-correct probe cuts false rejection from 58-92% to at most 5%, but keeps only 5-39% of suites. Repairing them instead, by rewriting each wrong expected value to what a reference execution returns, raises yield 3.3-10.6x across four author families.
分享
阅读原文 ↗