Learning to Foresee: Unveiling the Unlocking Efficiency of On-Policy Distillation
学习预见:揭示在线蒸馏的解锁效率
Yuchen Cai, Ding Cao, Liang Lin, Chunxi Luo, Xin Xu, Kai Yang, Weijie Liu, Saiyong Yang, Tianxiang Zhao, Guangzhong Sun, Guiquan Liu, Junfeng Fang
AI总结 本文研究了在线蒸馏(OPD)的效率来源,提出EffOPD方法通过适应性选择 extrapolation 步长和沿当前更新方向移动来加速OPD,实现了3倍的训练加速同时保持最终性能。
详情
在线蒸馏(OPD)已成为大型语言模型的一种高效的后训练范式。然而,现有研究大多将其优势归因于更密集和稳定的监督,而OPD效率背后的参数级机制仍不清晰。本文认为OPD的效率源于一种“预见”机制:它在训练早期就建立了指向最终模型的稳定更新轨迹。这种预见体现在两个方面。首先,在模块分配层面,OPD识别出边际效用低的区域,并将更新集中在对推理更关键的模块上。其次,在更新方向层面,OPD表现出更强的低秩集中,其主导子空间在训练早期就与最终更新子空间紧密对齐。基于这些发现,我们提出了EffOPD,一种即插即用的加速方法,通过自适应选择extrapolation步长并沿当前更新方向移动来加速OPD。EffOPD不需要额外可训练模块或复杂的超参数调优,实现了平均3倍的训练加速,同时保持可比的最终性能。整体而言,我们的发现为理解OPD的效率提供了参数动态视角,并为设计更高效的大型语言模型后训练方法提供了实用见解。
On-policy distillation (OPD) has emerged as an efficient post-training paradigm for large language models. However, existing studies largely attribute this advantage to denser and more stable supervision, while the parameter-level mechanisms underlying OPD's efficiency remain poorly understood. In this work, we argue that OPD's efficiency stems from a form of ``foresight'': it establishes a stable update trajectory toward the final model early in training. This foresight manifests in two aspects. First, at the \textbf{Module-Allocation Level}, OPD identifies regions with low marginal utility and concentrates updates on modules that are more critical to reasoning. Second, at the \textbf{Update-Direction Level}, OPD exhibits stronger low-rank concentration, with its dominant subspaces aligning closely with the final update subspace early in training. Building on these findings, we propose \textbf{EffOPD}, a plug-and-play acceleration method that speeds up OPD by adaptively selecting an extrapolation step size and moving along the current update direction. EffOPD requires no additional trainable modules or complex hyperparameter tuning, and achieves an average training acceleration of $3\times$ while maintaining comparable final performance. Overall, our findings provide a parameter-dynamics perspective for understanding the efficiency of OPD and offer practical insights for designing more efficient post-training methods for large language models.