smevals - a small eval suite for evaluating models, prompts, and harnesses

<p><strong><a href="https://primeradiant.com/blog/2026/smevals.html">smevals - a small eval suite for evaluating models, prompts, and harnesses</a></strong></p> I've been working with Jesse Vincent's <a href="https://primeradiant.com">Prime Radiant</a> applied AI research lab building out this evals framework to help answer questions about the capabilities of different models.</p> <p>The result is <strong><a href="https://github.com/prime-radiant-inc/smevals">smevals</a></strong>, a new tool for running small eval suites across different model configurations and grading the results.</p> <p>The <a href="https://primeradiant.com/blog/2026/smevals.html">blog entry</a> describes the tool in detail. Here's the 10 second version:</p> <ol> <li>Tell your coding agent to <code>run uvx smevals docs</code> to learn the tool (this outputs <a href="https://github.com/prime-radiant-inc/smevals/blob/main/README.md">the README</a>)</li> <li>Then tell it to build you an eval suite</li> </ol> <p>Once you've created an eval - which takes the form of a directory with some YAML files - you can run it against models like this:</p> <pre><code>uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6 </code></pre> <p>Runs are treated separately from grading operations - you can grade your runs (against your defined set of checks) using:</p> <pre><code>uvx smevals grade path-to-eval/ </code></pre> <p>Then you can run a localhost web server to explore the results:</p> <pre><code>uvx smevals serve path-to-eval/ </code></pre> <p>Or run the <code>smevals build</code> command to build that report as static HTML, which you can then host anywhere. Here's <a href="https://static.simonwillison.net/static/2026/smevals-haiku-build/#/haiku">an example</a> showing an eval suite I built to evaluate how well models can write haikus.</p> <p><img alt="Screenshot of an evaluation dashboard for a haiku-writing benchmark, testing whether models can reply with exactly three non-empty lines. A header describes the eval, with panels below showing a leaderboard ranking three GPT models by score, lists of recent runs and recent grades, tag pass rates, the two haiku prompts that were tested, and details of the graders used with a 0.8 pass threshold." src="https://static.simonwillison.net/static/2026/smevals-report.webp" /></p> <p>The most time-consuming part of this project was figuring out the vocabulary for it! Here's what I settled on, quoted from the announcement:</p> <blockquote> <ul> <li>An <strong>eval</strong> is a collection of challenges designed to answer a question about a model, for example, how good is that model at generating SVGs?</li> <li>Each eval is a collection of <strong>tasks</strong>. A task is a specific challenge, for example "Generate an SVG of a pelican riding a bicycle".</li> <li>When you run the eval you do so against one or more <strong>configs</strong>. Each config specifies a model to be evaluated, but may also include other parameters to test, such as different system prompts, model parameters, or agent harnesses.</li> <li>A <strong>run</strong> records what happened when a specific config was used to execute a specific task. A <strong>runner</strong> is the script that executes a run.</li> <li>Once you have collected one or more runs, you need to evaluate the results to see how well the model (or config) did. This is done by a <strong>grader</strong>, which produces a <strong>grade</strong>.</li> <li>Each grader runs a sequence of <strong>checks</strong>. These can be simple operations, like checking for a specific string in the output, or confirming that the output is valid XML. They can also be more complicated custom operations (implemented as scripts called <strong>checkers</strong>), including using other models to answer questions about the run.</li> </ul> </blockquote> <p>I've been trying to figure out an approach I like for evals for several years now. <code>smevals</code> is my third iteration on the idea and it feels right to me. I