Your smallest local model might be your best one - I measured 4 of mine

在AI大模型竞赛中，一个反常识的结论出现了：模型小，可能反而更厉害。一位开发者在自己的笔记本电脑上实测了四个本地模型，结果最迷你的1.1GB模型在算术和代码任务上拿了第一，而体积最大的7B模型排名垫底。这说明模型的能力和体积大小不能直接画等号，小模型用对了场景，效率完全能碾压大块头。这次测试还有一个关键发现：如果评测方法不严谨，结果可能完全失真。比如模型会先输出思考过程再给答案，或者答完题还喜欢额外解释几句，这些都会干扰自动评分，让明明正确的答案被判成零分。好在作者把测试工具开源了，大家都可以拿它来验证自己手头的模型。这个结论对普通人最直接的影响是：以后用AI应用可能更省钱、更流畅。开发者可以根据任务类型灵活选配模型，简单活儿交给轻量模型，复杂推理才动用大模型，既保证体验又降低运营成本。同时也提醒我们，不要太迷信大厂宣传的"参数越大越聪明"，实测数据比营销话术靠谱得多。

<p>I had seven models sitting in Ollama and no idea which one to use for what. So I stopped guessing and measured.</p> <p>152 generations on a 16 GB laptop. Greedy decoding, deterministic grading - exact number, exact string, JSON field, regex. No LLM judge, so there is no second model's bias to audit.</p> <h2> The result </h2> <p>No model won every category.</p> <div class="table-wrapper-paragraph"><table> <thead> <tr> <th>task type</th> <th>deepseek-r1:1.5b (1.1 GB)</th> <th>llama3.2:3b (2.0 GB)</th> <th>gemma:2b</th> <th>codellama 7b</th> </tr> </thead> <tbody> <tr> <td>arithmetic (12)</td> <td><strong>10/12</strong></td> <td>2/12</td> <td>2/12</td> <td>3/12</td> </tr> <tr> <td>extraction (9)</td> <td>4/9</td> <td><strong>9/9</strong></td> <td>7/9</td> <td>8/9</td> </tr> <tr> <td>classification (8)</td> <td>6/8</td> <td><strong>7/8</strong></td> <td><strong>7/8</strong></td> <td>2/8</td> </tr> <tr> <td>code output (6)</td> <td><strong>5/6</strong></td> <td>3/6</td> <td>1/6</td> <td>4/6</td> </tr> <tr> <td>median latency</td> <td>14.6 s</td> <td>1.6 s</td> <td>2.7 s</td> <td>6.3 s</td> </tr> </tbody> </table></div> <p>The 1.1 GB reasoning model is five times better at arithmetic than the 2 GB general model - and less than half as good at structured extraction, at nine times the latency. The smallest model I own scored highest overall. The largest came third.</p> <p>Size did not predict accuracy.</p> <h2> One thing that nearly broke the measurement </h2> <p>Reasoning models emit a <code><think></code> block before answering. My first grader read the whole response, so it kept scoring the model's intermediate working instead of its final answer - marking correct answers wrong. Stripping reasoning blocks and anchoring on the final answer changed deepseek's score dramatically.</p> <p>A second bug: gemma answers "Negative." then explains itself unprompted. I was only checking the last line, so a correct answer scored zero. That would have penalised every chatty small model across the whole benchmark.</p> <p>Both are now covered by tests. If you build evals, check this before you trust your numbers.</p> <h2> Caveats, up front </h2> <p>3-12 prompts per category, so a single item moves a percentage a lot. Only auto-gradable short-answer tasks, which excludes the open-ended writing bigger models are usually better at. One machine, Q4 quants, backend-reported timings with an uncontrolled OS cache.</p> <h2> Run it on your own models </h2> <p>MIT, runs entirely locally, never pulls or deletes models:</p> <p><a href="https://github.com/RitikPatill/routefoundry" rel="noopener noreferrer">https://github.com/RitikPatill/routefoundry</a></p> <p>Raw rows and conditions are committed in the repo, so you can check the grading rather than trust the table.</p>