How Much Is One Recurrence Worth? Iso-Depth Scaling Laws for Looped Language Models
循环一次值有多大?循环语言模型的等深度缩放定律
机构 * Chair for AI in Healthcare and Medicine, Technical University of Munich(慕尼黑技术大学人工智能在医疗和医学中的主任) ; Department of Computing, Imperial College London(伦敦帝国理工学院计算机系) ; Munich Center for Machine Learning (MCML), Germany(慕尼黑机器学习中心(MCML)) ; Hasso Plattner Institute for Digital Engineering, University of Potsdam, Germany(波茨坦大学数字工程霍普夫研究所)
AI总结 研究通过等深度预训练测量循环变换器中一次循环的价值,推导出缩放定律并发现循环等价指数φ=0.46,表明共享循环比独特块更差,展示了φ作为诊断工具的实用性。
Comments v3: substantially refined framing + minor corrections v2: added case studies on truncated-BPTT and hyperconnections