When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation

核心亮点：一项系统研究首次量化了AI评测基准的“饱和”现象，发现近半数基准已无法再有效区分模型优劣。通俗解读：AI的能力测试就像学生考试，题目出得再难，练多了大家都能考满分，这时候考试就失去了选拔意义。这篇论文分析了60个语言模型基准测试，结果发现大约一半已经“考糊了”，尤其是老基准，几乎区分不出新模型的差别。有趣的是，研究发现基准是否抗饱和，关键不在于测试数据是否公开，而在于是否有专家持续人工维护和更新题目。就像教材再好，也得靠老师不断出新卷子，才能避免学生刷题刷出高分。实际影响：以后我们看到的AI跑分可能会越来越少，因为分数普遍“通货膨胀”。这项研究提醒开发者和评测机构，不能只看旧榜单，而要建设动态、由专家精心设计的评测体系。对普通用户而言，这意味着未来选择AI助手时，不能迷信“分数高”，更应关注实际体验；而对整个行业来说，这推动着AI评估从“应试教育”转向“素质教育”，让技术真正朝着有用的方向进步，而不是刷分刷出来的虚高成绩。

Skip to main content arXiv is now an independent nonprofit! Learn more × Search Submit Donate Log in Search arXiv Press Enter to search · Advanced search Computer Science > Artificial Intelligence arXiv:2602.16763 (cs) [Submitted on 18 Feb 2026 (v1), last revised 29 Jun 2026 (this version, v3)] Title:When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation Authors:Mubashara Akhtar, Anka Reuel, Prajna Soni, Sanchit Ahuja, Pawan Sasanka Ammanamanchi, Ruchit Rawal, Vilém Zouhar, Srishti Yadav, Chenxi Whitehouse, Dayeon Ki, Jennifer Mickel, Leshem Choshen, Marek Šuppa, Jan Batzner, Jenny Chim, Jeba Sania, Yanan Long, Hossein A. Rahmani, Christina Knight, Yiyang Nan, Jyoutir Raj, Yu Fan, Shubham Singh, Subramanyam Sahoo, Eliya Habba, Usman Gohar, Siddhesh Pawar, Robert Scholz, Arjun Subramonian, Jingwei Ni, Mykel Kochenderfer, Sanmi Koyejo, Mrinmaya Sachan, Stella Biderman, Zeerak Talat, Avijit Ghosh, Irene Solaiman View a PDF of the paper titled When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation, by Mubashara Akhtar and 36 other authors View PDF HTML (experimental) Abstract:Artificial intelligence benchmarks are an important mechanism for measuring model progress and guiding deployment decisions. However, benchmarks quickly "saturate", making it difficult to differentiate models and diminishing their long-term value. In this study, we define benchmark saturation and analyze it across 60 language model benchmarks using 14 properties that relate to saturation. We find that nearly half of the our benchmarks exhibit saturation, with rates increasing with age. Further, we find that resilience to saturation is impacted by expert-curation, not by public test data. Our results suggest that design choices can extend benchmark longevity and inform more durable evaluation approaches. Comments: Accepted at ICML 2026 Subjects: Artificial Intelligence (cs.AI) Cite as: arXiv:2602.16763 [cs.AI] (or arXiv:2602.16763v3 [cs.AI] for this version) https://doi.org/10.48550/arXiv.2602.16763 Focus to learn more arXiv-issued DOI via DataCite Submission history From: Mubashara Akhtar [view email] [v1] Wed, 18 Feb 2026 16:51:37 UTC (222 KB) [v2] Sat, 30 May 2026 16:41:50 UTC (640 KB) [v3] Mon, 29 Jun 2026 17:01:58 UTC (636 KB) Full-text links: Access Paper: View a PDF of the paper titled When AI Benchmarks Plateau: A Systematic Study of Benchmark Saturation, by Mubashara Akhtar and 36 other authorsView PDFHTML (experimental)TeX Source view license Current browse context: cs.AI < prev | next > new | recent | 2026-02 Change to browse by: cs References & Citations NASA ADSGoogle Scholar Semantic Scholar export BibTeX citation Loading... BibTeX formatted citation × loading... Data provided by: Bookmark Bibli