When benchmark inferences do not compose: Projectibility in AI evaluation

AI 中文解读

核心亮点：一项研究发现，AI测试结果就像积木，单块再结实，随便拼在一起也可能散架——单个环节的证据都有效，组合起来却可能站不住脚。通俗解读：过去我们觉得，AI在某个测试里表现好，再把它用在别处也应该靠谱。但这篇论文指出，每个测试步骤之间可能存在“接口错位”：测试里的任务和实际场景不是一回事，或者测试数据偷偷“共享”过，导致看似独立的证据其实互相牵连。就好比一个人数学考满分，不代表他买菜算账就快，中间缺了“应用”这一步。研究者提出，要像审计账目一样检查AI评估链条，确保每一步的衔接都严丝合缝。实际影响：以后AI公司在宣称“我们通过了安全测试”时，就不能再拿单个测试结果当万能通行证了。普通用户也会受益——比如医疗、招聘或法律咨询里用的AI，评估流程会更严格，避免因为“测试看着不错”就盲目上马，最终减少AI在真实场景里掉链子的风险。简单说，这是给AI的“简历”加了道核实环节，防止夸大宣传。

arXiv:2607.26159v1 Announce Type: new Abstract: An AI benchmark result rarely reaches a consequential claim in one step. Evaluators generalize it to further cases, interpret it as evidence of capability, extrapolate it to new tasks, transport it to another system or site, and combine it with assumptions about human review and downstream consequences. Validity-centred approaches require evidence for each claim. This paper identifies a further epistemic problem: warranted links don't automatically make a warranted chain. The target of one study may not be the source of the next; system, population, outcome, or conditions may change at the interface; and shared data or model lineage may make apparently independent support dependent. Projectibility concerns whether a bounded extension from observed to unobserved cases is warranted. Goodman supplies the problem of rival extensions; argument-based validity supplies an architecture for testing them. The paper's distinctive claim is a non-composition principle: support for adjacent projections warrants their composition only when endpoints and assumptions align and dependence and uncertainty are carried through. A legal-research case shows how benchmark evidence and a deployment study can each be sound while remaining parallel. A reanalysis and simulation show why aggregate stability can erase distinctions a later projection requires. The resulting projectibility audit diagnoses unsupported joins in benchmark-to-use arguments.