"The Whole Is Greater Than the Sum of Its Parts": A Compatibility-Aware Multi-Teacher CoT Distillation Framework
整体大于部分之和:一种兼容性感知的多教师CoT蒸馏框架
机构 * State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(人机混合增强智能国家重点实验室,人工智能与机器人研究院,西安交通大学) ; Nankai University(南开大学) ; The Hong Kong University of Science and Technology(Guangzhou)(香港科技大学(广州)) ; School of Software Engineering, State Key Laboratory of Human-Machine Hybrid Augmented Intelligence, Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University(软件学院,人机混合增强智能国家重点实验室,人工智能与机器人研究院,西安交通大学)
专题命中 逻辑推理 :CoT(title,title_cn);reasoning(abstract);chain-of-thought(abstract);分类 cs.CL、cs.AI
AI总结 本文提出COMPACT框架,通过动态加权不同教师的梯度,结合多维指标提升学生模型的推理能力,有效整合多样化推理能力并减少灾难性遗忘。
Comments 11pages, 9figures