语言大模型 / LLM

LongMagpie: A Self-synthesis Method for Generating Large-scale Long-context Instructions

Chaochen Gao, Xing Wu, Zijia Lin, Debing Zhang, Songlin Hu

机构 * Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所） ； School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院） ； Xiaohongshu Inc（小红书公司） ； Tsinghua University（清华大学） 纠错

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI)

Yan Li, Tianyi Zhang, Zechuan Li, Soyeon Caren Han

机构 * The University of Sydney（悉尼大学） ； The University of Melbourne（墨尔本大学） ； Hunan University（湖南大学） 纠错

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 9 pages, under review in the conference

Writing Like the Best: Exemplar-Based Expository Text Generation

Yuxiang Liu, Kevin Chen-Chuan Chang

机构 * University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） 纠错

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ACL 2025. Camera-ready version

Boosting Long-Context Management via Query-Guided Activation Refilling

Hongjin Qian, Zheng Liu, Peitian Zhang, Zhicheng Dou, Defu Lian

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments ACL25 Main Conference

Training Long-Context LLMs Efficiently via Chunk-wise Optimization

Wenhao Li, Yuxin Zhang, Gen Luo, Daohai Yu, Rongrong Ji

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funneling

Zefan Cai, Yichi Zhang, Bofei Gao, Yuliang Liu, Yucheng Li, Tianyu Liu, Keming Lu, Wayne Xiong, Yue Dong, Junjie Hu, Wen Xiao

机构 * University of Wisconsin - Madison（威斯康星大学麦迪逊分校） ； Peking University（北京大学） ； Nanjing University（南京大学） ； University of Surrey（Surrey 大学） ； Qwen（通义实验室） ； University of California - Riverside（加州大学河滨分校） ； Microsoft（微软公司） 纠错

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Putting It All into Context: Simplifying Agents with LCLMs

Mingjian Jiang, Yangjun Ruan, Luis Lastras, Pavan Kapanipathi, Tatsunori Hashimoto

机构 * Stanford University（斯坦福大学） ； IBM Research（IBM研究院） ； University of Toronto（多伦多大学） 纠错

专题命中 长上下文与记忆 ：language model(abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Round and Round We Go! What makes Rotary Positional Encodings useful?

Federico Barbero, Alex Vitvitskyi, Christos Perivolaropoulos, Razvan Pascanu, Petar Veličković

机构 * University of Oxford（牛津大学） ； Google DeepMind（谷歌DeepMind） 纠错

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Mixture of Sparse Attention: Content-Based Learnable Sparse Attention via Expert-Choice Routing

Piotr Piękos, Róbert Csordás, Jürgen Schmidhuber

机构 * KAUST, AI Initiative（KAUST人工智能计划） ； Stanford University（斯坦福大学） 纠错

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Memorization and Knowledge Injection in Gated LLMs

Xu Pan, Ely Hahami, Zechen Zhang, Haim Sompolinsky

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation

Yi Lu, Wanxu Zhao, Xin Zhou, Chenxin An, Chenglong Wang, Shuo Li, Yuming Yang, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang

机构 * Fudan University（复旦大学） ； The University of Hong Kong（香港大学） ； Northeastern University（东北大学） 纠错

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

LiveLongBench: Tackling Long-Context Understanding for Spoken Texts from Live Streams

Yongxuan Wu, Runyu Chen, Peiyu Liu, Hongjin Qian

机构 * University of International Business and Economics（国际商务经济大学） ； Beijing Academy of Artificial Intelligence（北京人工智能研究院） 纠错

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

MOOSComp: Improving Lightweight Long-Context Compressor via Mitigating Over-Smoothing and Incorporating Outlier Scores

Fengwei Zhou, Jiafei Song, Wenjin Jason Li, Gengjian Xue, Zhikang Zhao, Yichao Lu, Bailin Na

机构 * OPPO CTG 纠错

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

HEMA : A Hippocampus-Inspired Extended Memory Architecture for Long-Context AI Conversations

Kwangseob Ahn

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Does Spatial Cognition Emerge in Frontier Models?

Santhosh Kumar Ramakrishnan, Erik Wijmans, Philipp Kraehenbuehl, Vladlen Koltun

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Published in ICLR 2025

It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimization

Ali Behrouz, Meisam Razaviyayn, Peilin Zhong, Vahab Mirrokni

专题命中 长上下文与记忆 ：language model(abstract)；foundation model(abstract)；分类 cs.AI、cs.LG

Moving Beyond Next-Token Prediction: Transformers are Context-Sensitive Language Generators

Phill Kyu Rhee

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 11 pages, 2 figures

MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentation

Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Defu Lian, Zhicheng Dou, Tiejun Huang

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments theWebConf 2025. Codes and models are in https://github.com/qhjqhj00/MemoRAG

Differential Transformer

Tianzhu Ye, Li Dong, Yuqing Xia, Yutao Sun, Yi Zhu, Gao Huang, Furu Wei

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted as an Oral Presentation at ICLR 2025

BigO(Bench) -- Can LLMs Generate Code with Controlled Time and Space Complexity?

Pierre Chambon, Baptiste Roziere, Benoit Sagot, Gabriel Synnaeve

专题命中 长上下文与记忆 ：LLM(abstract)；language model(abstract)；分类 cs.CL、cs.AI

PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention

Lida Chen, Dong Xu, Chenxin An, Xintao Wang, Yikai Zhang, Jiangjie Chen, Zujie Liang, Feng Wei, Jiaqing Liang, Yanghua Xiao, Wei Wang

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments for associated code, see https://github.com/w568w/PowerAttention

CoKV: Optimizing KV Cache Allocation via Cooperative Game

Qiheng Sun, Hongwei Zhang, Haocheng Xia, Jiayao Zhang, Jinfei Liu, Kui Ren

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Activation-aware Probe-Query: Effective Key-Value Retrieval for Long-Context LLMs Inference

Qingfa Xiao, Jiachuan Wang, Haoyang Li, Cheng Deng, Jiaqi Tang, Shuangyin Li, Yongqi Zhang, Jun Wang, Lei Chen

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

PolarQuant: Quantizing KV Caches with Polar Transformation

Insu Han, Praneeth Kacham, Amin Karbasi, Vahab Mirrokni, Amir Zandieh

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge

Daniel Tamayo, Aitor Gonzalez-Agirre, Javier Hernando, Marta Villegas

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Journal ref Findings of the Association for Computational Linguistics: ACL 2024. Pages: 5831-5847

From Few to Many: Self-Improving Many-Shot Reasoners Through Iterative Optimization and Generation

Xingchen Wan, Han Zhou, Ruoxi Sun, Hootan Nakhost, Ke Jiang, Sercan Ö. Arık

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Expanded version of the ICLR 2025 paper

Retrieve, Summarize, Plan: Advancing Multi-hop Question Answering with an Iterative Approach

Zhouyu Jiang, Mengshu Sun, Lei Liang, Zhiqiang Zhang

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted by WWW2025 Agent4IR Workshop

Doc-Guided Sent2Sent++: A Sent2Sent++ Agent with Doc-Guided memory for Document-level Machine Translation

Jiaxin Guo, Yuanchang Luo, Daimeng Wei, Ling Zhang, Zongyao Li, Hengchao Shang, Zhiqiang Rao, Shaojun Li, Jinlong Yang, Zhanglin Wu, Hao Yang

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Extract Information from Hybrid Long Documents Leveraging LLMs: A Framework and Dataset

Chongjian Yue, Xinrun Xu, Xiaojun Ma, Lun Du, Zhiming Ding, Shi Han, Dongmei Zhang, Qi Zhang

专题命中 长上下文与记忆 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments ICASSP 2025

FocusLLM: Precise Understanding of Long Context by Dynamic Condensing

Zhenyu Li, Yike Zhang, Tengyu Pan, Yutao Sun, Zhichao Duan, Junjie Fang, Rong Han, Zixuan Wang, Jianyong Wang

专题命中 长上下文与记忆 ：LLM(abstract)；language model(abstract)；分类 cs.CL、cs.AI