语言大模型 / LLM

SoftMCL: Soft Momentum Contrastive Learning for Fine-grained Sentiment-aware Pre-training

Jin Wang, Liang-Chih Yu, Xuejie Zhang

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

Comments Accepted by LREC-COLING 2024

Memory Traces: Are Transformers Tulving Machines?

Jean-Marie Chauvet

专题命中 长上下文与记忆 ：foundation model(abstract)；分类 cs.AI

Comments 14 pages, 1 figure and 4 tables

Learning a Fourier Transform for Linear Relative Positional Encodings in Transformers

Krzysztof Marcin Choromanski, Shanda Li, Valerii Likhosherstov, Kumar Avinava Dubey, Shengjie Luo, Di He, Yiming Yang, Tamas Sarlos, Thomas Weingarten, Adrian Weller

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.LG

Comments AISTATS 2024

Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings

Gaifan Zhang, Yi Zhou, Danushka Bollegala

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

PolySketchFormer: Fast Transformers via Sketching Polynomial Kernels

Praneeth Kacham, Vahab Mirrokni, Peilin Zhong

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.LG

Comments Added results of more experiments. Added a link to our JAX implementation of models

Towards Robustness and Diversity: Continual Learning in Dialog Generation with Text-Mixup and Batch Nuclear-Norm Maximization

Zihan Wang, Jiayu Xiao, Mengxiang Li, Zhongjiang He, Yongxiang Li, Chao Wang, Shuangyong Song

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

Comments 12 pages, 2 figures

Processing Natural Language on Embedded Devices: How Well Do Transformer Models Perform?

Souvika Sarkar, Mohammad Fakhruddin Babar, Md Mahadi Hassan, Monowar Hasan, Shubhra Kanti Karmaker Santu

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

Journal ref ICPE 2024

Functional Interpolation for Relative Positions Improves Long Context Transformers

Shanda Li, Chong You, Guru Guruganesh, Joshua Ainslie, Santiago Ontanon, Manzil Zaheer, Sumit Sanghai, Yiming Yang, Sanjiv Kumar, Srinadh Bhojanapalli

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.LG

Comments 26 pages; ICLR 2024 camera ready version

Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers

Abhimanyu Rajeshkumar Bambhaniya, Amir Yazdanbakhsh, Suvinay Subramanian, Sheng-Chun Kao, Shivani Agrawal, Utku Evci, Tushar Krishna

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.LG

Comments 18 pages, 8 figures, 17 tables. Code is available at https://github.com/abhibambhaniya/progressive_gradient_flow_nm_sparsity

Memory-Inspired Temporal Prompt Interaction for Text-Image Classification

Xinyao Yu, Hao Sun, Ziwei Niu, Rui Qin, Zhenjia Bai, Yen-Wei Chen, Lanfen Lin

专题命中 长上下文与记忆 ：foundation model(abstract)；分类 cs.AI

RanPAC: Random Projections and Pre-trained Models for Continual Learning

Mark D. McDonnell, Dong Gong, Amin Parveneh, Ehsan Abbasnejad, Anton van den Hengel

专题命中 长上下文与记忆 ：foundation model(abstract)；分类 cs.LG

Comments 32 pages, 11 figures

Journal ref 37th Annual Conference on Neural Information Processing Systems (NeurIPS 2023), Dec 2023, New Orleans, United States

PoET: A generative model of protein families as sequences-of-sequences

Timothy F. Truong, Tristan Bepler

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.LG

Journal ref Advances in Neural Information Processing Systems (Vol. 36), 2023

Unsupervised Continual Anomaly Detection with Contrastively-learned Prompt

Jiaqi Liu, Kai Wu, Qiang Nie, Ying Chen, Bin-Bin Gao, Yong Liu, Jinbao Wang, Chengjie Wang, Feng Zheng

专题命中 长上下文与记忆 ：prompting(abstract)；分类 cs.LG

Comments Accepted by AAAI 2024

Linear Attention via Orthogonal Memory

Jun Zhang, Shuyang Jiang, Jiangtao Feng, Lin Zheng, Lingpeng Kong

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

Comments 16 pages

VISIT: Visualizing and Interpreting the Semantic Information Flow of Transformers

Shahar Katz, Yonatan Belinkov

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

Comments EMNLP Findings 2023

Non-Sequential Ensemble Kalman Filtering using Distributed Arrays

Cédric Travelletti, Jörg Franke, David Ginsbourger, Stefan Brönnimann

专题命中 长上下文与记忆 ：prompting(abstract)；分类 cs.LG

Leveraging Cross-Utterance Context For ASR Decoding

Robert Flynn, Anton Ragni

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

ARC-NLP at PAN 2023: Hierarchical Long Text Classification for Trigger Detection

Umitcan Sahin, Izzet Emre Kucukkaya, Cagri Toraman

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

Comments Accepted by PAN at CLEF 2023

FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning

Tri Dao

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.LG

Extending the Frontier of ChatGPT: Code Generation and Debugging

Fardin Ahsan Sakib, Saadat Hasan Khan, A. H. M. Rezaul Karim

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

Constant Memory Attention Block

Leo Feng, Frederick Tung, Hossein Hajimirsadeghi, Yoshua Bengio, Mohamed Osama Ahmed

专题命中 长上下文与记忆 ：foundation model(abstract)；分类 cs.LG

Comments Workshop version of arXiv:2305.14567

AVScan2Vec: Feature Learning on Antivirus Scan Data for Production-Scale Malware Corpora

Robert J. Joyce, Tirth Patel, Charles Nicholas, Edward Raff

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.LG

Small Character Models Match Large Word Models for Autocomplete Under Memory Constraints

Ganesh Jawahar, Subhabrata Mukherjee, Debadeepta Dey, Muhammad Abdul-Mageed, Laks V. S. Lakshmanan, Caio Cesar Teodoro Mendes, Gustavo Henrique de Rosa, Shital Shah

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

Comments SustaiNLP 2023

Adapting Learned Sparse Retrieval for Long Documents

Thong Nguyen, Sean MacAvaney, Andrew Yates

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.LG

Comments SIGIR 2023

Journal ref SIGIR 2023

Revisiting Parallel Context Windows: A Frustratingly Simple Alternative and Chain-of-Thought Deterioration

Kejuan Yang, Xiao Liu, Kaiwen Men, Aohan Zeng, Yuxiao Dong, Jie Tang

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers

Lili Yu, Dániel Simig, Colin Flaherty, Armen Aghajanyan, Luke Zettlemoyer, Mike Lewis

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.LG

KALM: Knowledge-Aware Integration of Local, Document, and Global Contexts for Long Document Understanding

Shangbin Feng, Zhaoxuan Tan, Wenqian Zhang, Zhenyu Lei, Yulia Tsvetkov

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

Comments ACL 2023

REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory

Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A. Ross, Alireza Fathi

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI

Comments Published on CVPR 2023

SEAL: Semantic Frame Execution And Localization for Perceiving Afforded Robot Actions

Cameron Kisailus, Daksh Narang, Matthew Shannon, Odest Chadwicke Jenkins

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.AI

Comments Currently in review for IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2023). Contact: kisailus@umich.edu

The Surprising Computational Power of Nondeterministic Stack RNNs

Brian DuSell, David Chiang

专题命中 长上下文与记忆 ：language model(abstract)；分类 cs.CL

Comments 21 pages, 8 figures. Published at ICLR 2023