SAAG: Structured Agent Assessment and Grounding

AI 中文解读

SAAG这项新技术给AI智能体调用装上了一个“诊断放大镜”，能分步骤精准找出错误原因，而不是简单打个对错标记。目前评估AI调用工具或服务时，通常只给出一个“通过”或“不通过”的分数。但AI可能选对了工具却瞎编了参数，或者参数看似合理但选错了工具——这些不同的失败原因都被同一个分数掩盖了，开发者根本不知道问题出在哪。SAAG就像医生做体检一样，把评估拆成三个环节：先看AI有没有选对工具，再看参数结构是否完整，最后核对参数值是否准确。每个环节都能给出具体反馈，还能指导AI自己修复错误，比如参数不对就只修正参数，而不需要重头再来。这项技术最直接的影响是让AI助手变得更可靠。以后你在智能音箱里命令“帮我定个餐厅”，或者让手机助手查询天气、操控家电，AI不仅出错率会降低，就算出错也能自我纠正，不用你反复解释。对于开发者来说，分析AI行为变得更轻松，能更快优化产品的可靠性。长远看，AI与真实世界交互的准确性会明显提升，让“数字管家”真正值得信赖。

arXiv:2607.18245v1 Announce Type: new Abstract: Exact-match evaluation of agent-calling obscures qualitatively different failure modes: a model may select the right function yet hallucinate argument values, or satisfy a schema while choosing a agent for the wrong reason. Existing benchmarks collapse these distinctions into a single binary score, leaving practitioners unable to diagnose where agent calls fail. We propose SAAG a cascaded diagnostic framework that decomposes agent-calling evaluation into three sequential stages: registry conformance, structural completeness, and argument grounding, each producing interpretable stage-specific diagnostics. These diagnostics additionally enable iterative self-repair: on prediction failure, the stage-specific signal guides targeted correction without leaking ground-truth values. We evaluate this framework on a controlled benchmark derived from Glaive's function-calling dataset across registry sizes of 5, 10, and 15 agents using three local sub-4B-parameter models. Structured feedback consistently improves argument precision and reduces value hallucination relative to single-pass inference and uninformative binary feedback, while end-to-end F1 gains are modest and model-dependent. These results suggest that stage-decomposed diagnostic evaluation is a necessary lens for understanding and improving agent-calling reliability across model families and registry scales.