One QK Channel, Many Sources: Guarding Low-Precision Attention Collapse
一个QK通道,多个源:防范低精度注意力崩溃
Shuxiao Xie, Shuyang Xie, Yuan Cao, Dezhi Ran, Wei Yang, Tao Xie
机构
*
Beijing Tongming Lake Information Technology Application Innovation Center (TLAIC)(北京通明湖信息技术应用创新中心(TLAIC))
;
Fudan University Institute of Systems for Advanced Computing(复旦大学先进计算系统研究所)
;
Harbin Institute of Technology(哈尔滨工业大学)
;
Peking University(北京大学)
;
Shanghai Institute of Systems for Open Computing(上海开放计算系统研究所)
AdaHAT: Adaptive Hard Attention to the Task in Task-Incremental Learning
AdaHAT:任务增量学习中的自适应硬注意力机制
Pengxiang Wang, Hongbo Bo, Jun Hong, Weiru Liu, Kedian Mu
机构
*
School of Mathematical Sciences, Peking University(北京大学数学科学学院)
;
Population Health Sciences Institute, Newcastle University(纽卡斯尔大学人口健康科学研究所)
;
School of Engineering Mathematics and Technology, University of Bristol(布里斯托尔大学工程数学与技术学院)
;
School of Computing and Creative Technologies, University of the West of England(西英格兰大学计算与创意技术学院)
机构
*
University of Electronic Science and Technology of China(电子科技大学)
;
Tibet University(西藏大学)
;
Peking University(北京大学)
;
Southwest Jiaotong University(西南交通大学)
机构
*
City University of Hong Kong(香港城市大学)
;
Huawei Technologies(华为技术有限公司)
;
Mohamed bin Zayed University of Artificial Intelligence(穆罕默德·本·扎耶德人工智能大学)
;
McGill University(麦吉尔大学)
;
Peking University(北京大学)
机构
*
Peking University(北京大学)
;
University of Illinois at Urbana-Champaign(伊利诺伊大学厄巴纳-香槟分校)
;
University of Pennsylvania(宾夕法尼亚大学)
;
Purdue University(普渡大学)
;
Stanford University(斯坦福大学)
机构
*
Peking University(北京大学)
;
University of California, Los Angeles(加州大学洛杉矶分校)
;
Arena
;
University of Maryland(马里兰大学)
;
Mohamed bin Zayed University of Artificial Intelligence(莫莫迪 bin Zayed 人工智能大学)
GaiaFlow: Semantic-Guided Diffusion Tuning for Carbon-Frugal Search
GaiaFlow:基于语义的扩散调优用于碳节约搜索
Rong Fu, Jia Yee Tan, Chunlei Meng, Shuo Yin, Xiaowen Ma, Wangyu Wu, Simon Fong
机构
*
University of Macau(澳门大学)
;
Renmin University of China(中国人民大学)
;
Fudan University(复旦大学)
;
Tsinghua University(清华大学)
;
Zhejiang University(浙江大学)
;
University of Liverpool(利物浦大学)
;
Peking University(北京大学)
;
Northeast Normal University(东北师范大学)
;
Minzu University of China(民族大学)
TempoNet: Slack-Quantized Transformer-Guided Reinforcement Scheduler for Adaptive Deadline-Centric Real-Time Dispatchs
TempoNet:基于Transformer的强化学习调度器用于自适应截止时间导向的实时调度
Rong Fu, Yibo Meng, Zeyu Zhang, Ziming Guo, Jia Yee Tan, Xiaojing Du, Simon James Fong
机构
*
University of Macau(澳门大学)
;
Tsinghua University(清华大学)
;
Northeast Normal University(东北师范大学)
;
Shanghai AI Laboratory(上海人工智能实验室)
;
The Australian National University(澳大利亚国立大学)
;
Peking University(北京大学)
;
Harbin University of Science and Technology(哈尔滨理工大学)
;
Renmin University of China(中国人民大学)
;
Adelaide University(阿德莱德大学)
Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewards
尾感知信息论泛化用于RLHF和SGLD
Huiming Zhang, Binghan Li, Wan Tian, Qiang Sun
机构
*
Institute of Artificial Intelligence, Beihang University(北京航空航天大学人工智能研究院)
;
Beijing Advanced Innovation Center for Future Blockchain and Privacy Computing(北京未来区块链与隐私计算高精尖创新中心)
;
Advanced Institute of Information Technology, Peking University(北京大学信息技术高等研究院)
;
Wangxuan Institute of Computer Technology, Peking University(北京大学王选计算机技术研究所)
;
Computer and Mathematical Sciences, Computer Science, and Statistics, University of Toronto(多伦多大学计算机与数学科学、计算机科学和统计学系)
;
MBZUAI(穆罕默德·本·扎耶德人工智能大学)
机构
*
National Key Laboratory of Big Data and Decision(大数据与决策国家重点实验室)
;
National University of Defense Technology(国防科技大学)
;
The Center for machine learning research(机器学习研究中心)
;
Peking University(北京大学)
;
Tsinghua University(清华大学)
;
The Hong Kong University of Science and Technology(香港科学与技术大学)
Ganghui Cao, Shimin Wang, Martin Guay, Jinzhi Wang, Zhisheng Duan, Marios M. Polycarpou
机构
*
State Key Laboratory for Turbulence and Complex Systems, Department of Mechanics and Engineering Science, College of Engineering, Peking University(湍流与复杂系统国家重点实验室,力学与工程科学系,工程学院,北京大学)
;
Massachusetts Institute of Technology(麻省理工学院)
;
Queen’s University(皇后大学)
;
KIOS Research and Innovation Center of Excellence and the Department of Electrical and Computer Engineering, University of Cyprus(科斯卓越研究中心与创新中心,塞浦路斯大学电气与计算机工程系)
机构
*
Peking University(北京大学)
;
School of Advanced Manufacturing and Robotics, Peking University(北京大学先进制造与机器人学院)
;
Shanghai Jiao Tong University(上海交通大学)
;
Southern University of Science and Technology(南方科技大学)
机构
*
Shanghai University of Finance and Economics(上海财经大学)
;
Meituan(美团)
;
The Chinese University of Hong Kong, Shenzhen(香港中文大学(深圳))
;
Peking University(北京大学)