语言大模型 / LLM

M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, Mohit Bansal

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Comments Project webpage: https://m3docrag.github.io

SAM 2: Segment Anything in Images and Videos

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, Eric Mintun, Junting Pan, Kalyan Vasudev Alwala, Nicolas Carion, Chao-Yuan Wu, Ross Girshick, Piotr Dollár, Christoph Feichtenhofer

专题命中 长上下文与记忆 ：foundation model(abstract)；分类 cs.AI、cs.LG

Comments Website: https://ai.meta.com/sam2

On the token distance modeling ability of higher RoPE attention dimension

Xiangyu Hong, Che Jiang, Biqing Qi, Fandong Meng, Mo Yu, Bowen Zhou, Jie Zhou

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2024 Findings

MoEUT: Mixture-of-Experts Universal Transformers

Róbert Csordás, Kazuki Irie, Jürgen Schmidhuber, Christopher Potts, Christopher D. Manning

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

Comments Accepted to NeurIPS 2024

Automatic Summarization of Long Documents

Naman Chhibbar, Jugal Kalita

专题命中 长上下文与记忆 ：LLM(abstract)；分类 cs.CL、cs.AI

Comments 9 pages (including bibliography) with 6 figures. ACL 2023 proceedings format

AnaloBench: Benchmarking the Identification of Abstract and Long-context Analogies

Xiao Ye, Andrew Wang, Jacob Choi, Yining Lu, Shreya Sharma, Lingfeng Shen, Vijay Tiyyala, Nicholas Andrews, Daniel Khashabi

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2024 (Main)

SwitchHead: Accelerating Transformers with Mixture-of-Experts Attention

Róbert Csordás, Piotr Piękos, Kazuki Irie, Jürgen Schmidhuber

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted to NeurIPS 2024

A Controlled Study on Long Context Extension and Generalization in LLMs

Yi Lu, Jing Nathan Yan, Songlin Yang, Justin T. Chiu, Siyu Ren, Fei Yuan, Wenting Zhao, Zhiyong Wu, Alexander M. Rush

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

ReMEmbR: Building and Reasoning Over Long-Horizon Spatio-Temporal Memory for Robot Navigation

Abrar Anwar, John Welsh, Joydeep Biswas, Soha Pouya, Yan Chang

专题命中 长上下文与记忆 ：LLM(abstract)；分类 cs.CL、cs.AI

Benchmarking General-Purpose In-Context Learning

Fan Wang, Chuan Lin, Yang Cao, Yu Kang

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

Great Memory, Shallow Reasoning: Limits of $k$NN-LMs

Shangyi Geng, Wenting Zhao, Alexander M Rush

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

xLSTMTime : Long-term Time Series Forecasting With xLSTM

Musleh Alharthi, Ausif Mahmood

专题命中 长上下文与记忆 ：prompting(abstract)；分类 cs.AI、cs.LG

Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms

Tian Meng, Yang Tao, Wuliang Yin

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

Scaling Granite Code Models to 128K Context

Matt Stallone, Vaibhav Saxena, Leonid Karlinsky, Bridget McGinn, Tim Bula, Mayank Mishra, Adriana Meza Soria, Gaoyuan Zhang, Aditya Prasad, Yikang Shen, Saptha Surendran, Shanmukha Guttula, Hima Patel, Parameswaran Selvam, Xuan-Hong Dang, Yan Koyfman, Atin Sood, Rogerio Feris, Nirmit Desai, David D. Cox, Ruchir Puri, Rameswar Panda

专题命中 长上下文与记忆 ：pretraining(abstract)；分类 cs.CL、cs.AI

Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture

Sangjun Park, JinYeong Bak

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

Comments ICML 2024 Spotlight. 29 pages, 15 figures, 11 tables

Journal ref Proceedings of the 41st International Conference on Machine Learning, PMLR 235:39587-39615, 2024

AutoGRAMS: Autonomous Graphical Agent Modeling Software

Ben Krause, Lucia Chen, Emmanuel Kahembwe

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Bifurcated Attention: Accelerating Massively Parallel Decoding with Shared Prefixes in LLMs

Ben Athiwaratkun, Sujan Kumar Gonugondla, Sanjay Krishna Gouda, Haifeng Qian, Hantian Ding, Qing Sun, Jun Wang, Jiacheng Guo, Liangfu Chen, Parminder Bhatia, Ramesh Nallapati, Sudipta Sengupta, Bing Xiang

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

On the Power of Convolution Augmented Transformer

Mingchen Li, Xuechen Zhang, Yixiao Huang, Samet Oymak

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

M5: A Whole Genome Bacterial Encoder at Single Nucleotide Resolution

Agust Egilsson

专题命中 长上下文与记忆 ：foundation model(abstract)；分类 cs.AI、cs.LG

Comments 13 pages, 5 figures

Promoting Exploration in Memory-Augmented Adam using Critical Momenta

Pranshu Malviya, Gonçalo Mordido, Aristide Baratin, Reza Babanezhad Harikandeh, Jerry Huang, Simon Lacoste-Julien, Razvan Pascanu, Sarath Chandar

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI、cs.LG

Comments Published in Transactions on Machine Learning Research

MATTER: Memory-Augmented Transformer Using Heterogeneous Knowledge Sources

Dongkyu Lee, Chandana Satya Prakash, Jack FitzGerald, Jens Lehmann

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Comments ACL2024-Findings

Discrete Prompt Compression with Reinforcement Learning

Hoyoun Jung, Kyung-Joong Kim

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Reward-based Input Construction for Cross-document Relation Extraction

Byeonghu Na, Suhyeon Jo, Yeongmin Kim, Il-Chul Moon

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted at ACL 2024 main conference

Are Long-LLMs A Necessity For Long-Context Tasks?

Hongjin Qian, Zheng Liu, Peitian Zhang, Kelong Mao, Yujia Zhou, Xu Chen, Zhicheng Dou

专题命中 长上下文与记忆 ：LLM(abstract)；分类 cs.CL、cs.AI

Comments 18 pages

Correlation Dimension of Natural Language in a Statistical Manifold

Xin Du, Kumiko Tanaka-Ishii

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Comments Published at Physical Review Research

Journal ref Physical Review Research, 6(2), L022028 (2024)

Evaluating the Efficacy of AI Techniques in Textual Anonymization: A Comparative Study

Dimitris Asimopoulos, Ilias Siniosoglou, Vasileios Argyriou, Sotirios K. Goudos, Konstantinos E. Psannis, Nikoleta Karditsioti, Theocharis Saoulidis, Panagiotis Sarigiannidis

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.AI

Adaptive Memory Replay for Continual Learning

James Seale Smith, Lazar Valkov, Shaunak Halbe, Vyshnavi Gutta, Rogerio Feris, Zsolt Kira, Leonid Karlinsky

专题命中 长上下文与记忆 ：foundation model(abstract)；分类 cs.CL、cs.LG

Comments CVPR-W 2024 (Spotlight)

ChatPattern: Layout Pattern Customization via Natural Language

Zixiao Wang, Yunheng Shen, Xufeng Yao, Wenqian Zhao, Yang Bai, Farzan Farnia, Bei Yu

专题命中 长上下文与记忆 ：LLM(abstract)；分类 cs.CL、cs.AI

Comments Accepted by DAC24

SEA: Sparse Linear Attention with Estimated Attention Mask

Heejun Lee, Jina Kim, Jeffrey Willette, Sung Ju Hwang

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL、cs.LG

Comments 9 main pages

COCOA: CBT-based Conversational Counseling Agent using Memory Specialized in Cognitive Distortions and Dynamic Prompt

Suyeon Lee, Jieun Kang, Harim Kim, Kyoung-Mee Chung, Dongha Lee, Jinyoung Yeo

专题命中 长上下文与记忆 ：prompting(abstract)；分类 cs.CL、cs.AI

Comments 4 pages, 2 figures