语言大模型 / LLM

From Language Models over Tokens to Language Models over Characters

Tim Vieira, Ben LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Brian DuSell, John Terilla, Timothy J. O'Donnell, Ryan Cotterell

机构 * City University of New York（纽约城市大学） ； McGill University（麦吉尔大学） ； Canada CIFAR AI Chair（加拿大CIFAR人工智能主席） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments ICML 2025

Language Models over Canonical Byte-Pair Encodings

Tim Vieira, Tianyu Liu, Clemente Pasti, Yahya Emara, Brian DuSell, Benjamin LeBrun, Mario Giulianelli, Juan Luis Gastaldi, Timothy J. O'Donnell, Ryan Cotterell

机构 * McGill University（麦吉尔大学） ； Canada CIFAR AI Chair（加拿大 CIFAR 人工智能主席） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments ICML 2025

PreGIP: Watermarking the Pretraining of Graph Neural Networks for Deep Intellectual Property Protection

Enyan Dai, Minhua Lin, Suhang Wang

机构 * Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）） ； The Pennsylvania State University（宾夕法尼亚州立大学） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Structure-Aware Fill-in-the-Middle Pretraining for Code

Linyuan Gong, Alvin Cheung, Mostafa Elhoushi, Sida Wang

机构 * UC Berkeley（加州大学伯克利分校） ； FAIR at Meta（Meta旗下的FAIR） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.AI

Comments 14 pages

Multilingual Encoder Knows more than You Realize: Shared Weights Pretraining for Extremely Low-Resource Languages

Zeli Su, Ziyin Zhang, Guixian Xu, Jianing Liu, XU Han, Ting Zhang, Yushuang Dong

机构 * Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE（民族语言智能分析与安全治理重点实验室） ； Minzu University of China（民族大学） ； Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 预训练与数据 ：pretraining(title)；language model(abstract)；分类 cs.CL、cs.AI

Comments ACL 2025 camera-ready

One Model for One Graph: A New Perspective for Pretraining with Cross-domain Graphs

Jingzhe Liu, Haitao Mao, Zhikai Chen, Bingheng Li, Wenqi Fan, Mingxuan Ju, Tong Zhao, Neil Shah, Jiliang Tang

机构 * Michigan State University（密歇根州立大学） ； Hong Kong Polytechnic University（香港理工大学） ； Snap Inc.（Snap公司） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

A Lightweight Method to Disrupt Memorized Sequences in LLM

Parjanya Prajakta Prashant, Kaustubh Ponkshe, Babak Salimi

专题命中 预训练与数据 ：LLM(title)；language model(abstract)；分类 cs.CL、cs.LG

Comments 26 pages, 3 figures

SALMONN-omni: A Standalone Speech LLM without Codec Injection for Full-duplex Conversation

Wenyi Yu, Siyin Wang, Xiaoyu Yang, Xianzhao Chen, Xiaohai Tian, Jun Zhang, Guangzhi Sun, Lu Lu, Yuxuan Wang, Chao Zhang

机构 * Tsinghua University（清华大学） ； ByteDance（字节跳动） ； University of Cambridge（剑桥大学） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；分类 cs.CL、cs.AI

Dataless Knowledge Fusion by Merging Weights of Language Models

Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, Pengxiang Cheng

机构 * University of Southern California（南加州大学） ； Bloomberg（布莱尔） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments ICLR 2023; The code is available at https://github.com/bloomberg/dataless-model-merging and https://github.com/AuCson/RegMean-LLama3-8B. Fixed typos

Gender Trouble in Language Models: An Empirical Audit Guided by Gender Performativity Theory

Franziska Sofia Hafner, Ana Valdivia, Luc Rocher

机构 * University of Oxford（牛津大学） ； University of Oxford Institute of Advanced Studies (UCL)（牛津大学高级研究院） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

Journal ref FAccT '25: Proceedings of the 2025 ACM Conference on Fairness, Accountability, and Transparency

CMLFormer: A Dual Decoder Transformer with Switching Point Learning for Code-Mixed Language Modeling

Aditeya Baral, Allen George Ajith, Roshan Nayak, Mrityunjay Abhijeet Bhanja

机构 * Courant Institute of Mathematical Sciences（Courant数学科学研究所） ； Tandon School of Engineering（Tandon工程学院） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Achieving Tokenizer Flexibility in Language Models through Heuristic Adaptation and Supertoken Learning

Shaurya Sharthak, Vinayak Pahalwan, Adithya Kamath, Adarsh Shirawalmath

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

When Foundation Model Meets Federated Learning: Motivations, Challenges, and Future Directions

Weiming Zhuang, Chen Chen, Jingtao Li, Chaochao Chen, Yaochu Jin, Lingjuan Lyu

机构 * Sony Research（索尼研究） 纠错

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

The advantages of context specific language models: the case of the Erasmian Language Model

João Gonçalves, Nick Jelicic, Michele Murgia, Evert Stamhuis

机构 * Erasmus University Rotterdam（埃拉斯马大学鹿特丹分校） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments 12 pages, 3 figures, 1 table

DEPT: Decoupled Embeddings for Pre-training Language Models

Alex Iacob, Lorenzo Sani, Meghdad Kurmanji, William F. Shen, Xinchi Qiu, Dongqi Cai, Yan Gao, Nicholas D. Lane

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Published as a conference paper at ICLR 2025

Pretraining with random noise for uncertainty calibration

Jeonghwan Cheon, Se-Bum Paik

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

RankCLIP: Ranking-Consistent Language-Image Pretraining

Yiming Zhang, Zhuokai Zhao, Zhaorun Chen, Zhili Feng, Zenghui Ding, Yining Sun

专题命中 预训练与数据 ：pretraining(title)；language model(abstract)；分类 cs.AI、cs.LG

Comments Code and model checkpoints are available at https://github.com/Jam1ezhang/RankCLIP

Towards Neural Scaling Laws for Time Series Foundation Models

Qingren Yao, Chao-Han Huck Yang, Renhe Jiang, Yuxuan Liang, Ming Jin, Shirui Pan

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted by the 13th International Conference on Learning Representations (ICLR 2025)

NavRAG: Generating User Demand Instructions for Embodied Navigation through Retrieval-Augmented LLM

Zihan Wang, Yaohui Zhu, Gim Hee Lee, Yachun Fan

专题命中 预训练与数据 ：LLM(title,abstract)；分类 cs.CL、cs.AI

RDT-1B: a Diffusion Foundation Model for Bimanual Manipulation

Songming Liu, Lingxuan Wu, Bangguo Li, Hengkai Tan, Huayu Chen, Zhengyi Wang, Ke Xu, Hang Su, Jun Zhu

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

Comments 10 pages, conference

Time-MoE: Billion-Scale Time Series Foundation Models with Mixture of Experts

Xiaoming Shi, Shiyu Wang, Yuqi Nie, Dianqi Li, Zhou Ye, Qingsong Wen, Ming Jin

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted by the 13th International Conference on Learning Representations (ICLR 2025)

Movie Gen: A Cast of Media Foundation Models

Adam Polyak, Amit Zohar, Andrew Brown, Andros Tjandra, Animesh Sinha, Ann Lee, Apoorv Vyas, Bowen Shi, Chih-Yao Ma, Ching-Yao Chuang, David Yan, Dhruv Choudhary, Dingkang Wang, Geet Sethi, Guan Pang, Haoyu Ma, Ishan Misra, Ji Hou, Jialiang Wang, Kiran Jagadeesh, Kunpeng Li, Luxin Zhang, Mannat Singh, Mary Williamson, Matt Le, Matthew Yu, Mitesh Kumar Singh, Peizhao Zhang, Peter Vajda, Quentin Duval, Rohit Girdhar, Roshan Sumbaly, Sai Saketh Rambhatla, Sam Tsai, Samaneh Azadi, Samyak Datta, Sanyuan Chen, Sean Bell, Sharadh Ramaswamy, Shelly Sheynin, Siddharth Bhattacharya, Simran Motwani, Tao Xu, Tianhe Li, Tingbo Hou, Wei-Ning Hsu, Xi Yin, Xiaoliang Dai, Yaniv Taigman, Yaqiao Luo, Yen-Cheng Liu, Yi-Chiao Wu, Yue Zhao, Yuval Kirstain, Zecheng He, Zijian He, Albert Pumarola, Ali Thabet, Artsiom Sanakoyeu, Arun Mallya, Baishan Guo, Boris Araya, Breena Kerr, Carleigh Wood, Ce Liu, Cen Peng, Dimitry Vengertsev, Edgar Schonfeld, Elliot Blanchard, Felix Juefei-Xu, Fraylie Nord, Jeff Liang, John Hoffman, Jonas Kohler, Kaolin Fire, Karthik Sivakumar, Lawrence Chen, Licheng Yu, Luya Gao, Markos Georgopoulos, Rashel Moritz, Sara K. Sampson, Shikai Li, Simone Parmeggiani, Steve Fine, Tara Fowler, Vladan Petrovic, Yuming Du

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

A Novel Multi-Task Teacher-Student Architecture with Self-Supervised Pretraining for 48-Hour Vasoactive-Inotropic Trend Analysis in Sepsis Mortality Prediction

Houji Jin, Negin Ashrafi, Kamiar Alaei, Elham Pishgar, Greg Placencia, Maryam Pishgar

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Training Language Models on Synthetic Edit Sequences Improves Code Synthesis

Ulyana Piterbarg, Lerrel Pinto, Rob Fergus

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments ICLR 2025

Accessing Vision Foundation Models via ImageNet-1K

Yitian Zhang, Xu Ma, Yue Bai, Huan Wang, Yun Fu

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted by ICLR2025

A Group Symmetric Stochastic Differential Equation Model for Molecule Multi-modal Pretraining

Shengchao Liu, Weitao Du, Zhiming Ma, Hongyu Guo, Jian Tang

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Vision-Language Model Dialog Games for Self-Improvement

Ksenia Konyushkova, Christos Kaplanis, Serkan Cabi, Misha Denil

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.AI、cs.LG

Soup-of-Experts: Pretraining Specialist Models via Parameters Averaging

Pierre Ablin, Angelos Katharopoulos, Skyler Seto, David Grangier

专题命中 预训练与数据 ：pretraining(title)；language model(abstract)；分类 cs.CL、cs.LG

NSmark: Null Space Based Black-box Watermarking Defense Framework for Language Models

Haodong Zhao, Jinming Hu, Peixuan Li, Fangqi Li, Jinrui Sha, Tianjie Ju, Peixuan Chen, Zhuosheng Zhang, Gongshen Liu

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments In progress

ViT-2SPN: Vision Transformer-based Dual-Stream Self-Supervised Pretraining Networks for Retinal OCT Classification

Mohammadreza Saraei, Igor Kozak, Eung-Joo Lee

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG