Nobody Evaluates the Evaluator

<blockquote> <p>Originally published on <a href="https://harshkedia.com/writing/nobody-evaluates-the-evaluator/" rel="noopener noreferrer">harshkedia.com</a>. Cross-posted here in full.</p> <p><strong>TL;DR</strong></p> <p>An AI eval score is two claims at once: that the model did the thing, and that your grader is right about whether it did. The industry argues the first and almost never tests the second. In 2026 frontier judges have gotten genuinely good, so the unreliability moved: the score now turns on how you word the rubric and what your harness does, not on the model. Here is why that happens, with numbers, and what to do instead of trusting the number.</p> </blockquote> <p>Earlier this year Anthropic published a number that should have landed harder than it did. A model, Opus 4.5, scored 42% on a benchmark called CORE-Bench. Then, without touching the model, they fixed some rigid grading, some ambiguous task specs, and some harness bugs, and the same model scored 95%. Fifty-three points. The model did not get smarter overnight. The ruler changed.</p> <p>That is the whole problem in one anecdote. We treat an eval score as a measurement of the model. It is actually the product of two things: the model's behaviour, and the instrument that judged it. And that instrument, more and more, is itself a model, an LLM-as-judge, or a benchmark harness with a grader buried inside it. Almost nobody measures the instrument.</p> <h2> What an LLM-as-a-judge score actually claims </h2> <p>Write it out and it is obvious. A passing score asserts two things: that the model produced an acceptable output, and that your grader is correct that it was acceptable. The first claim gets all the attention: leaderboards, launch posts, benchmark wars. The second gets almost none. Yet you cannot tell "the model regressed" from "the grader drifted" unless you have versioned and frozen the grader, and hardly anyone ships a grader with a version number. Your judge is a dependency that changes underneath you without a changelog.</p> <p><a href="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqt62ehyz0stdegzq1zx1.png" class="article-body-image-wrapper"><img src="https://media2.dev.to/dynamic/image/width=800%2Cheight=%2Cfit=scale-down%2Cgravity=auto%2Cformat=auto/https%3A%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Farticles%2Fqt62ehyz0stdegzq1zx1.png" alt="A model produces an answer that is scored by an instrument made of an LLM judge, a rubric, and a harness. The model path is what everyone measures; the instrument is unversioned and unvalidated, and nothing validates the validator." width="800" height="416"></a></p> <p><em>Fig 1 · a score is mostly a claim about the instrument that produced it, and the instrument is the part nobody measures.</em></p> <h2> The good news: LLM judges do agree with humans </h2> <p>Modern judges are good, and getting better, which ruins the easy hot take. The canonical study of LLM-as-judge found a strong model agreed with human graders more often than the humans agreed with each other.</p> <blockquote> <p>the agreement between GPT-4 and humans reaches 85%, which is even higher than the agreement among humans (81%).</p> <p>— Zheng et al., Judging LLM-as-a-Judge with MT-Bench (2023)</p> </blockquote> <p>I wanted to see how that holds up on today's models, so I ran a small probe: ten question-and-answer pairs, graded PASS or FAIL by the three newest Gemini judges at temperature zero, all on one shared rubric. They agreed on all ten. I ran the newest of them twelve times on each item, same input every time, and it never once contradicted itself. The old fear that judges are slot machines, that even temperature zero flips verdicts, is largely obsolete on frontier models.</p> <p><em>three current judges, one rubric, temperature 0</em><br> </p> <div class=