The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation

AI 中文解读

核心亮点：这项研究揭示了一个被忽视的真相——评价AI编程助手时，光看模型名字和通过率远远不够，背后的“脚手架”系统才是决定成本和效率的关键。通俗解读：就像评价一个厨师不能只看菜谱，还得看厨房设备和操作流程。AI编程助手也是如此，除了AI模型本身，还有一套“脚手架”程序负责调用工具、管理信息、控制决策节奏。研究者发现，在不同的脚手架下，同样是完成一道编程题，消耗的计算资源最多能差40倍，而模型自身的通过率差异其实很小，甚至可以忽略不计。这意味着，平时大家关注的哪家AI更强，很可能只是脚手架不同带来的错觉。实际影响：对普通程序员或企业来说，挑选AI助手时别只盯着新模型名字，更应该关注它搭配的“脚手架”是否高效。选对了组合，既能保证解题质量，又能大幅降低使用成本、减少等待时间。未来，AI工具的评价标准可能会更全面，就像买车不仅要看发动机，还要看底盘和变速箱一样，帮助用户做出更明智的选择。

arXiv:2607.22585v1 Announce Type: new Abstract: Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3.6 Plus and MiniMax M2.5 across three open-source harnesses (Goose, OpenCode, OpenHands-SDK) on a stratified 50-task subset of Terminal-Bench Pro. Harness choice induces up to a 40x difference in tokens per solved task, while paired within-model pass-rate differences remain 0-8 percentage points (95% paired-task bootstrap CIs include zero except for the largest gap). Failure fingerprints replicate across models (REASON for Goose, VERIFY/MAX_TURNS for OpenHands-SDK, idle-loop/TIME for OpenCode), indicating harness-level biases that are largely model-independent. For human-centered coding-agent evaluation, model name alone is an incomplete comparison unit: harness-model pairs determine real-world cost, latency, and oversight burden; no-action turns are a per-task wait tax, not just a token tax. We therefore recommend selecting harness-model pairs by pass rate under token/latency budgets, and reporting token usage, latency, and full harness specifications alongside any model comparison. We release anonymized configs, raw trial logs, aggregated snapshots, and analysis scripts.