语言大模型 / LLM - arXivDaily 专题

2309.11568 2023-09-22 cs.AI cs.CL cs.LG 67%

BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model

Nolan Dey, Daria Soboleva, Faisal Al-Khateeb, Bowen Yang, Ribhu Pathria, Hemant Khachane, Shaheer Muhammad, Zhiming, Chen, Robert Myers, Jacob Robert Steeves, Natalia Vassilieva, Marvin Tom, Joel Hestness

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2309.08715 2023-09-19 cs.FL 67%

Formalizing BPE Tokenization

Martin Berglund, Brink van der Merwe

专题命中长上下文与记忆：large language model(abstract)；language model(abstract)

Comments In Proceedings NCMA 2023, arXiv:2309.07333

Journal ref EPTCS 388, 2023, pp. 16-27

2306.10231 2023-06-21 cs.CL cs.AI cs.LG 67%

GLIMMER: generalized late-interaction memory reranker

Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Sumit Sanghai, William W. Cohen, Joshua Ainslie

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2301.10448 2023-06-06 cs.CL cs.AI cs.LG 67%

Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute

Michiel de Jong, Yury Zemlyanskiy, Nicholas FitzGerald, Joshua Ainslie, Sumit Sanghai, Fei Sha, William Cohen

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICML 2023

2210.11807 2022-10-24 cs.CL cs.AI cs.LG 67%

Is Encoder-Decoder Redundant for Neural Machine Translation?

Yingbo Gao, Christian Herold, Zijian Yang, Hermann Ney

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments accepted at AACL2022

2206.14969 2022-07-01 cs.CL cs.AI cs.LG 67%

Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging?

Xiang Zhou, Shiyue Zhang, Mohit Bansal

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NAACL 2022 (16 pages)

2202.10447 2022-06-28 cs.LG cs.AI cs.CL cs.NE 67%

Transformer Quality in Linear Time

Weizhe Hua, Zihang Dai, Hanxiao Liu, Quoc V. Le

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to the 39th International Conference on Machine Learning (ICML'22)

2205.08343 2022-06-24 cs.IR 67%

Moving Stuff Around: A study on efficiency of moving documents into memory for Neural IR models

Arthur Câmara, Claudia Hauff

专题命中长上下文与记忆：large language model(abstract)；language model(abstract)

Comments 7 pages, 2 figures. Accepted to the ReNeuIR workshop at SIGIR 2022

1808.08720 2022-03-30 cs.LG cs.AI cs.CL stat.ML 67%

Predefined Sparseness in Recurrent Sequence Models

Thomas Demeester, Johannes Deleu, Fréderic Godin, Chris Develder

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments the SIGNLL Conference on Computational Natural Language Learning (CoNLL, 2018)

2203.08913 2022-03-18 cs.LG cs.AI cs.CL 67%

Memorizing Transformers

Yuhuai Wu, Markus N. Rabe, DeLesley Hutchins, Christian Szegedy

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published as a conference paper at ICLR 2022 (spotlight)

2105.07510 2021-10-12 cs.CL cs.AI cs.LG 67%

Doc2Dict: Information Extraction as Text Generation

Benjamin Townsend, Eamon Ito-Fisher, Lily Zhang, Madison May

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2110.02220 2021-10-08 eess.AS cs.AI cs.CL cs.LG cs.NE 67%

Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognition

Tsendsuren Munkhdalai, Khe Chai Sim, Angad Chandorkar, Fan Gao, Mason Chua, Trevor Strohman, Françoise Beaufays

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 5 pages, 3 figures, 3 tables

2012.08695 2020-12-17 cs.CL cs.AI cs.IR cs.LG 67%

DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition

Weizhou Shen, Junqing Chen, Xiaojun Quan, Zhixian Xie

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by AAAI 2021 main conference

2007.00849 2020-07-03 cs.CL cs.AI cs.LG 67%

Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge

Pat Verga, Haitian Sun, Livio Baldini Soares, William W. Cohen

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

1908.02367 2019-08-08 cs.CL cs.AI cs.LG 67%

Semantic Role Labeling with Associated Memory Network

Chaoyu Guan, Yuhao Cheng, Hai Zhao

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published at NAACL 2019; This is camera Ready version; Code is available at https://github.com/Frozenmad/AMN_SRL

1810.06682 2019-03-13 cs.LG cs.AI cs.CL stat.ML 67%

Trellis Networks for Sequence Modeling

Shaojie Bai, J. Zico Kolter, Vladlen Koltun

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published at ICLR 2019

1709.05027 2018-02-13 cs.LG cs.AI cs.CL cs.NE 67%

Learning Intrinsic Sparse Structures within Long Short-Term Memory

Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, Hai Li

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published in ICLR 2018 ( the Sixth International Conference on Learning Representations)

2512.23852 2026-01-01 cs.LG cs.CL 66%

Trellis: Learning to Compress Key-Value Memory in Attention Models

Trellis: 在注意力模型中学习压缩键值记忆

Mahdi Karami, Ali Behrouz, Praneeth Kacham, Vahab Mirrokni

机构 * Google Research（谷歌研究）

专题命中长上下文与记忆：language model(abstract,comments)；分类 cs.CL、cs.LG

AI总结 Trellis通过动态压缩键值内存提升注意力模型的效率与长上下文处理能力

Comments In Second Conference on Language Modeling (COLM) (2025)

2104.08401 2021-10-08 cs.CL cs.AI 66%

Enriching a Model's Notion of Belief using a Persistent Memory

Nora Kassner, Oyvind Tafjord, Hinrich Schutze, Peter Clark

专题命中长上下文与记忆：language model(abstract,comments)；分类 cs.CL、cs.AI

Comments This is an old and now obsolete draft. See arXiv:2109.14723 ("BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief") for the final paper

2608.19013 2026-08-20 cs.LG cs.AI 新提交 62%

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

利用工具链持续学习：超越模型参数的持续适应

Borui Kang, Jinrui Gu, Junhan Lv, Wenbin Li, Lei Wang, Yang Gao

机构 * University of Wollongong（卧龙岗大学）； Nanjing University（南京大学）

专题命中长上下文与记忆：foundation model(abstract)；分类 cs.AI、cs.LG

AI总结该研究提出工具链持续学习（HCL）范式，围绕冻结基础模型演化工具链，通过受保护演化机制缓解工具链级遗忘，在多任务实验中相对基准增益超10%，可调整稳定性-可塑性权衡。

2608.12597 2026-08-14 cs.LG cs.AI 新提交 62%

Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks

预测随机低维重参数化何时能训练神经网络

Andrew Cheng, Ali Eslamian, Jie Cheng, Mehdi Zargham, Qiang Cheng

机构 * Tsinghua University（清华大学）； The University of Manchester（曼彻斯特大学）； University of Kentucky（肯塔基大学）； Miami University（迈阿密大学）； University of Dayton（代顿大学）； Institute for Biomedical Informatics, University of Kentucky（肯塔基大学生物医学信息学研究所）

专题命中长上下文与记忆：language model(abstract)；分类 cs.AI、cs.LG

AI总结该研究针对随机低维重参数化训练神经网络的潜在空间规模问题，提出取向分辨二次主公式，引入RaMaN模型，实现内存优化，实验验证其预测与转换跟踪性能优于取向无关近似。

2607.19058 2026-08-14 cs.LG cs.AI 版本更新 62%

Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training

优化器状态应存于何处？内存高效的专家混合训练分层状态分配

Nuemaan Malik

专题命中长上下文与记忆：language model(abstract)；分类 cs.AI、cs.LG

AI总结研究MoE训练中优化器状态存储位置，提出SkewAdam方法，根据参数群体差异分层分配状态，大幅减少内存占用，提升训练效率，验证了优化器状态存储位置对训练效果的重要性。

Comments 12 pages, 4 figures, 9 tables. v2: adds Adam-mini discussion, learning-rate sweeps with repeated seeds for the AdamW and Adafactor baselines, and a tier ablation; corrects the attribution of the perplexity advantage between momentum and the factored estimator. Code and per-run training logs: https://github.com/nuemaan/skewadam

2608.11701 2026-08-13 cs.LG cs.AI 新提交 62%

Consolidator: Learning Persistent Routed Memory Across Context Boundaries

Consolidator：学习跨上下文边界的持久路由记忆

Sungwoo Goo, Hwi-yeol Yun, Sangkeun Jung

机构 * College of Pharmacy, Chungnam National University（忠南国立大学药学院）； Department of Computer Science & Engineering, Chungnam National University（忠南国立大学计算机科学与工程学院）

专题命中长上下文与记忆：pretraining(abstract)；分类 cs.AI、cs.LG

AI总结该研究提出Consolidator算子，在冻结骨干与记忆接口的情况下，通过少量可训练参数实现跨上下文边界的持久路由记忆，在两段式模10映射任务中显著提升了更新后映射的召回率。

2606.28876 2026-08-13 cs.CL cs.LG 版本更新 62%

MMLA: How Memory Lets the Past Shape the Future

内存管理的长上下文注意力：可编辑请求局部记忆的初步研究

Junyi Zou, Avrova Donz

机构 * Zjydiary（智纪日记）

专题命中长上下文与记忆：LLM(abstract)；分类 cs.CL、cs.LG

AI总结提出一种混合架构，将快速循环或稀疏骨干与可编辑的请求局部记忆槽和查询时稀疏回退相结合，以解决长上下文模型中的记忆写入、覆盖和抗污染问题。

Comments 16 pages, 11 figures, 5 tables. Substantially rewritten v3: retitled and reframed as MMLA; adds an eventized architecture, trusted row assembly, hard atomic overwrite/NULL, claim-evidence mapping, PM-I2 negative results, stop rules, and a conditional roadmap. Preserves the original Llama budget-gate failure. Project: https://github.com/MMLA-org/mmla-memory

2511.10696 2026-08-05 cs.CL cs.AI 版本更新 62%

$π$-Attention: Online Efficient Sparse Transformers for Long-Context Modeling

π-注意力：用于高效长上下文建模的周期性稀疏变换器

Pike D. Liu, Chang Liu, Yanxuan Yu

机构 * University of California - Los Angeles（加州大学洛杉矶分校）； Columbia University（哥伦比亚大学）

专题命中长上下文与记忆：language model(abstract)；分类 cs.CL、cs.AI

AI总结本文提出π注意力，通过周期性稀疏结构实现高效长上下文建模，相比环形注意力在接收场增长上更优，且在相同上下文长度下使用更少的GPU。

2608.00009 2026-08-04 cs.CL cs.AI 新提交 62%

AgentMemBench: A Systematic Benchmark for Evaluating Long-Term Memory Management Strategies in Conversational AI Agents

AgentMemBench：用于评估对话AI智能体长期记忆管理策略的系统基准

Ahmed Cherif

机构 * Sofrecom（索弗雷科姆）

专题命中长上下文与记忆：LLM(abstract)；分类 cs.CL、cs.AI

AI总结该研究构建了AgentMemBench基准，评估五种记忆策略及两款现有系统，发现外部键值存储（EKV）在长程对话记忆任务中表现最优，但存在内存占用成本，同时公开了全部可复现资源。

Comments 22 pages, 3 figures submitted on Neural Computing and Applications

2606.31650 2026-08-04 cs.LG cs.AI 版本更新 62%

ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL

ECHO: 在智能体强化学习中通过选择性回合记忆进行剪枝行动与追踪学习

Zijun Xie, Binbin Zheng, Enlei Gong, Jihua Liu, Yuyang You, Lingfeng Liu, Jiayao Tang, Guanqun Zhao, Aoqi Hu, Zeyu Chen

机构 * School of Mathematical Sciences, Peking University（北京大学数学科学学院）； Baidu Inc.（百度公司）； University of Science and Technology of China（中国科学技术大学）

专题命中长上下文与记忆：language agent(abstract)；分类 cs.AI、cs.LG

AI总结提出ECHO框架，通过选择性回合记忆和源索引重建解决长程智能体强化学习中的历史崩溃与可追踪学习问题，在BrowseComp-Plus上达到43.4%准确率，优于GRPO和SUPO。

2607.29459 2026-08-03 cs.LG cs.AI 新提交 62%

TFGformer: Multivariate Time Series Forecasting via Time-Frequency Graph Learning and Covariate Fusion

TFGformer：基于时频图学习与协变量融合的多变量时间序列预测

Yu Sun, Yuan Chang, Xiaohou Shi, Yan Sun

专题命中长上下文与记忆：foundation model(abstract)；分类 cs.AI、cs.LG

AI总结针对异构IoT传感器多变量时间序列预测问题，提出CrossRAG框架，结合SAM、FCC学习与CATF，在7个基准上性能优于仅参数化基线及现有检索增强预测方法。

2604.17121 2026-07-31 cs.LG cs.AI 版本更新 62%

The Topological Trouble With Transformers

Transformer 的拓扑困境

Michael C. Mozer, Shoaib Ahmed Siddiqui, Rosanne Liu

机构 * Google DeepMind（谷歌DeepMind）

专题命中长上下文与记忆：foundation model(abstract)；分类 cs.AI、cs.LG

AI总结本文探讨了Transformer在处理序列结构时的拓扑问题，指出其纯前馈架构限制了动态状态跟踪，提出应通过递归架构转向隐含激活动态，并介绍了连续思维Transformer架构的分类方法及未来研究方向。

2607.26523 2026-07-30 cs.LG cs.AI 新提交 62%

The Art of Not Forgetting A Local Learning Architecture for Continual Learning

不遗忘的艺术：面向持续学习的局部学习架构

Ashmith Atmuri, Yashaswini Rao Bhogarajula

机构 * Arkadhi Research（阿卡迪研究机构）

专题命中长上下文与记忆：language model(abstract)；分类 cs.AI、cs.LG

AI总结该研究提出持续学习架构CMP，采用稀疏关系编码、两层竞争记忆与局部更新，在语言建模实验中较参数匹配的Transformer基线表现出更低的灾难性遗忘，为持续学习提供了新方向。

Comments 28 pages, 11 figures

1. 长上下文与记忆 4789 篇

BTLM-3B-8K: 7B Parameter Performance in a 3B Parameter Model

Formalizing BPE Tokenization

GLIMMER: generalized late-interaction memory reranker

Pre-computed memory or on-the-fly encoding? A hybrid approach to retrieval augmentation makes the most of your compute

Is Encoder-Decoder Redundant for Neural Machine Translation?

Masked Part-Of-Speech Model: Does Modeling Long Context Help Unsupervised POS-tagging?

Transformer Quality in Linear Time

Moving Stuff Around: A study on efficiency of moving documents into memory for Neural IR models

Predefined Sparseness in Recurrent Sequence Models

Memorizing Transformers

Doc2Dict: Information Extraction as Text Generation

Fast Contextual Adaptation with Neural Associative Memory for On-Device Personalized Speech Recognition

DialogXL: All-in-One XLNet for Multi-Party Conversation Emotion Recognition

Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge

Semantic Role Labeling with Associated Memory Network

Trellis Networks for Sequence Modeling

Learning Intrinsic Sparse Structures within Long Short-Term Memory

Trellis: Learning to Compress Key-Value Memory in Attention Models

Enriching a Model's Notion of Belief using a Persistent Memory

Harness Continual Learning: Continual Adaptation Beyond Model Parameters

Predicting When Random Low-Dimensional Reparameterizations Train Neural Networks

Where Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training

Consolidator: Learning Persistent Routed Memory Across Context Boundaries

MMLA: How Memory Lets the Past Shape the Future

$π$-Attention: Online Efficient Sparse Transformers for Long-Context Modeling

AgentMemBench: A Systematic Benchmark for Evaluating Long-Term Memory Management Strategies in Conversational AI Agents

ECHO: Prune To Act, Trace To Learn With Selective Turn Memory In Agentic RL

TFGformer: Multivariate Time Series Forecasting via Time-Frequency Graph Learning and Covariate Fusion

The Topological Trouble With Transformers

The Art of Not Forgetting A Local Learning Architecture for Continual Learning