Taming Momentum: Rethinking Optimizer States Through Low-Rank Approximation
驯服动量:通过低秩近似重新思考优化器状态
机构 * University of Science and Technology of China(中国科学技术大学) ; NLPR & MAIS, Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所) ; School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院)
AI总结 LoRA-Pre通过低秩近似优化器状态,提升预训练和微调效率,实现内存节省与性能提升
Comments Camera-ready version. Accepted as Oral at ICLR 2026