Adaptive FastOPD: Progress-Aware Rollout Horizon Expansion for Efficient On-Policy Distillation

AI 中文解读

AI训练提速新方案来了！这项研究提出一种名为“自适应快速知识蒸馏”的技术，核心亮点在于它能像聪明的教练一样，根据学生AI的学习进度动态调整训练强度，不再死板地按固定时长“陪练”。通俗地说，以前AI向“老师AI”学习时，老师总是不厌其烦地全程指导，哪怕学生已经学会了还在重复讲，导致训练又慢又浪费。新方法会实时观察学生的表现，只有在发现学生卡壳、当前练习已充分掌握时，才让老师延长指导时间；如果学生已经学好，就果断停止，避免无意义的消耗。实测下来，训练速度提升49%到71%，而且最终AI的“成绩”还更好。对普通人来说，这意味着未来AI大模型的研发成本更低、迭代更快，你用到的智能助手、推荐系统或自动驾驶功能，可能会以更聪明、更懂你的姿态更早出现，同时背后的服务器能耗也减少了，可以说是又便宜又高效。

On-policy distillation (OPD) provides dense teacher supervision along student-generated trajectories, but its online rollout process incurs substantial computational cost, particularly when a few long responses delay batch completion. Existing acceleration methods typically control rollout length using fixed budgets or absolute teacher--student agreement thresholds, which may not reflect learning progress across different models and training stages. We propose Adaptive FastOPD, a progress-aware strategy that expands the rollout horizon only when learning near the current boundary region has plateaued and the current horizon is sufficiently utilized. The former is determined from four teacher--student signals measured relative to their values upon entering each horizon, making expansion responsive to stage-specific progress rather than a predefined step interval or an absolute threshold on the raw agreement signals, while the latter prevents a small number of long responses from triggering increases in rollout cost. Across two teacher--student pairs, Adaptive FastOPD achieves the highest average performance while reducing training time by 49.1--71.2\% relative to OPD 15K, and remains robust across a range of hyperparameter settings.