Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateaus
混合线性注意力大语言模型中的大规模激活:注意力前峰值与峰间平台期
Zunhai Su, Bohan Sun, Xialie Zhuang, Shuibai Zhang, He Xiao, Jing Xiong, Hengyuan Zhang, Zhongzhu Zhou, Tiantian Zhang, Ngai Wong, Chuan-Wei Kuo
机构
*
Startlux(星创公司)
;
Tsinghua University(清华大学)
;
University of Chinese Academy of Sciences(中国科学院大学)
;
The University of Hong Kong(香港大学)
;
University of Sydney(悉尼大学)
;
Columbia University(哥伦比亚大学)
Surfsvr: 2D Surface Priors as 3D Geometric Regularizers for Sparse Voxel Reconstruction
Surfsvr:将2D表面先验作为稀疏体素重建的3D几何正则化项
Yan Di, Chengxi Li, Yaoxing Wang, Mengge Liu, Zhigang Li, Ruida Zhang, Mingyang Li, Pengyuan Wang, Shan Gao, Xiangyang Ji
机构
*
HIT (Shenzhen)(哈尔滨工业大学(深圳))
;
Tsinghua University(清华大学)
;
NWPU(西北工业大学)
;
Beijing Institute of Control and Engineering(北京控制与工程研究所)
;
Technical University of Munich(慕尼黑工业大学)
机构
*
School of Artificial Intelligence, Beijing Normal University(北京师范大学人工智能学院)
;
Peking University(北京大学)
;
National Center for Applied Mathematics Shenzhen (NCAMS), Southern University of Science and Technology(南方科技大学深圳应用数学国家中心)
;
Department of Automation, Tsinghua University(清华大学自动化系)
;
National Research Council of Italy, Institute of Integrated Methodologies for Earth Observation (CNR-IMIOT)(意大利国家研究委员会综合地球观测方法研究所)
Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learning
漂移与依赖:基于重放的持续学习的分层信息论边界
Tieliang Gong, Zhongbo Zhang, Wen Wen, Yong-Jin Liu
机构
*
School of Computer Science and Technology, Xi’an Jiaotong University(西安交通大学计算机科学与技术学院)
;
Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系)