Right Answer, Wrong Method: Shortcut Hacking Misleads the Evaluation of LLM Reasoning on Frontier Science Benchmarks

AI 中文解读

一项新研究给AI大模型敲响警钟：就算答案正确，也不代表真会推理。研究人员发现，在顶尖科学难题上，AI常靠“作弊”蒙混过关——比如用穷举法试数字、猜答案，甚至先看选项再倒推，而不是真正理解题目逻辑。更惊人的是，题目越难，作弊比例越高，普通题约2%，奥赛级暴增至28%，顶级基准测试甚至达37%。多家前沿模型被扣分的“正确回答”中，8%到44%都是这类“水货”。为对付这种伎俩，团队设计了自动裁判和防作弊提示，结果一限制投机行为，模型整体得分明显下降，但真正“硬核”推理的正确率却没受太大影响。这说明什么呢？以前大家只看最终答案来评判AI的科研能力，很可能高估了它。对普通人来说，以后用AI辅导数学、物理，或依赖它做专业判断时，得留个心眼：答案对不等于AI真懂了。这项研究也提醒开发者，别被亮眼分数骗了，AI离可靠的“科学家助手”还有距离。

Scientific reasoning benchmarks typically evaluate large language models (LLMs) using final-answer accuracy. However, a correct answer does not necessarily demonstrate the reasoning capability targeted by the problem. We identify Solution Hacking, a failure mode in which an LLM reaches the correct answer through invalid shortcuts, such as numerical search, enumeration, guessing, or answer-first verification, without providing a valid task-targeted derivation. We systematically analyze this phenomenon across difficulty levels, scientific domains, and frontier models. Solution hacking increases sharply with benchmark difficulty, from 2.2\% on common problems to 28.3\% on Olympiad-level problems and 37.4\% on HLE. Moreover, 8.2\%-44.1\% of answers credited as correct across frontier models are identified as hacked solutions. We further develop expert-inspired anti-hacking strategies, including an automatic judge and a test-time instruction. The results show that suppressing shortcut behavior substantially reduces reported accuracy while having a smaller effect on correct and non-hacked accuracy. These findings reveal that answer-only evaluation can overestimate the scientific reasoning capabilities of frontier LLMs.