Structured Synthetic Reasoning Data for Arithmetic Fine-Tuning of Small Language Models

AI 中文解读

小语言模型也能搞定复杂算术题了！最新研究通过低成本方法，让原本“数学差”的小型AI在推理能力上大幅跃升，效果直追大模型。研究团队用GPT-5-mini生成了一套包含2万多道小学数学题的“结构化合成数据”，每道题不仅有自然语言解题步骤，还加入了苏格拉底式引导和故意干扰信息。然后他们在苹果M4芯片和16GB内存的普通电脑上，用LoRA技术微调了两个小模型（6亿和17亿参数）。结果令人惊喜：6亿参数的模型在小学数学题测试集上的准确率从36.5%提升到49.1%，17亿参数的模型更是从53.5%跃升至66.5%。微调后的模型不仅解题步骤更简洁，还更善于排除干扰、减少计算错误。这意味着，未来我们可能不需要联网调用大模型，就能在手机或笔记本上快速搞定数学问题。对于学生、教育软件开发者，以及任何想本地部署智能助手的用户来说，这无疑是个好消息——用小成本就能让AI变得“更聪明”。

arXiv:2607.18266v1 Announce Type: new Abstract: Small language models are attractive for local deployment, but they often struggle with multi-step arithmetic reasoning. We study whether structured synthetic reasoning data can improve this behaviour under consumer-hardware constraints. Starting from GSM8K, we generated a 21,250-example corpus of grade-school arithmetic word-problem variants using GPT-5-mini, combining natural-language solution traces, light Socratic-style cues, structural variation, and irrelevant distractor context. We then fine-tuned Qwen3-0.6B and Qwen3-1.7B with LoRA on consumer hardware (Apple M4, 16 GB RAM). Exact-match accuracy on GSM8K improved from 36.5% to 49.1% for Qwen3-0.6B and from 53.5% to 66.5% for Qwen3-1.7B. For Qwen3-1.7B, transfer to related arithmetic benchmarks was stronger, reaching 98.9% on MultiArith and 73.0% on SVAMP, compared with 54.4% and 45.3% for the base model. Qualitative analysis suggests that fine-tuned models produce shorter reasoning traces, make fewer arithmetic and distractor-use errors, and benefit more consistently from self-consistency sampling. These results show that low-cost synthetic data design can materially improve arithmetic adaptation in small language models. Because the intervention combines Socratic-style cues with other data-design choices, we interpret the gains as evidence for structured synthetic reasoning data rather than as a causal test of Socratic guidance alone.