语言大模型 / LLM

Self-Challenging Language Model Agents

Yifei Zhou, Sergey Levine, Jason Weston, Xian Li, Sainbayar Sukhbaatar

机构 * UC Berkeley（伯克利大学） ； FAIR at Meta（Meta 公司人工智能研究部） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Scaling Laws for Forgetting during Finetuning with Pretraining Data Injection

Louis Bethune, David Grangier, Dan Busbridge, Eleonora Gualdoni, Marco Cuturi, Pierre Ablin

机构 * Apple（苹果公司） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments 19 pages, 15 figures, preprint

FLAME-MoE: A Transparent End-to-End Research Platform for Mixture-of-Experts Language Models

Hao Kang, Zichun Yu, Chenyan Xiong

机构 * Language Technologies Institute（语言技术研究所） ； School of Computer Science（计算机科学学院） ； Foundation and Language Model Center（基础与语言模型中心） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.LG

Comments All code, training logs, and model checkpoints are available at https://github.com/cmu-flame/FLAME-MoE

How Syntax Specialization Emerges in Language Models

Xufeng Duan, Zhaoqian Yao, Yunhao Zhang, Shaonan Wang, Zhenguang G. Cai

机构 * Department of Linguistics and Modern Languages, The Chinese University of Hong Kong（香港中文大学语言学与现代语言系） ； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） ； Brain and Mind Institute, The Chinese University of Hong Kong（香港中文大学脑科学与心灵研究所） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

TabICL: A Tabular Foundation Model for In-Context Learning on Large Data

Jingang Qu, David Holzmüller, Gaël Varoquaux, Marine Le Morvan

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments Published at ICML 2025

Unified World Models: Coupling Video and Action Diffusion for Pretraining on Large Robotic Datasets

Chuning Zhu, Raymond Yu, Siyuan Feng, Benjamin Burchfiel, Paarth Shah, Abhishek Gupta

专题命中 预训练与数据 ：pretraining(title,abstract)；foundation model(abstract)；分类 cs.AI、cs.LG

Extractive Structures Learned in Pretraining Enable Generalization on Finetuned Facts

Jiahai Feng, Stuart Russell, Jacob Steinhardt

专题命中 预训练与数据 ：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Large Wireless Localization Model (LWLM): A Foundation Model for Positioning in 6G Networks

Guangjin Pan, Kaixuan Huang, Hui Chen, Shunqing Zhang, Christian Häger, Henk Wymeersch

机构 * Department of Electrical Engineering, Chalmers University of Technology（楚德大学电气工程系） ； School of Communication and Information Engineering, Shanghai University（上海大学通信与信息工程学院） ； Chalmers AI Research Center Consortium (CHAIR)（楚德人工智能研究中心联盟） ； National Academic Infrastructure for Supercomputing in Sweden (NAISS)（瑞典国家学术超级计算基础设施） ； SNS JU project 6G-DISAC under the EU’s Horizon Europe research and innovation Program（欧盟地平线欧洲研究与创新计划下的SNS JU项目6G-DISAC） ； Swedish Foundation for Strategic Research (SSF)（瑞典战略研究基金会） ； Chalmers Areas of Advance in ICT and Transport（楚德信息与交通领域先进领域） 纠错

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments 13 pages,16 figures.This work has been submitted to the IEEE for possible publication

Latent Action Pretraining from Videos

Seonghyeon Ye, Joel Jang, Byeongguk Jeon, Sejune Joo, Jianwei Yang, Baolin Peng, Ajay Mandlekar, Reuben Tan, Yu-Wei Chao, Bill Yuchen Lin, Lars Liden, Kimin Lee, Jianfeng Gao, Luke Zettlemoyer, Dieter Fox, Minjoon Seo

机构 * KAIST（韩国科学技术院） ； University of Washington（华盛顿大学） ； Microsoft Research（微软研究院） ； NVIDIA（英伟达） ； Allen Institute for AI（人工智能算法研究所） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；foundation model(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2025 Website: https://latentactionpretraining.github.io

Talking Heads: Understanding Inter-layer Communication in Transformer Language Models

Jack Merullo, Carsten Eickhoff, Ellie Pavlick

机构 * Department of Computer Science（计算机科学系） ； Brown University（布朗大学） ； School of Medicine（医学院） ； University of Tübingen（图宾根大学） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments Neurips 2024

NeMo-Inspector: A Visualization Tool for LLM Generation Analysis

Daria Gitman, Igor Gitman, Evelina Bakhturina

机构 * NVIDIA Corporation（英伟达公司） 纠错

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Presented at the NAACL 2025 conference

Beyond Public Access in LLM Pre-Training Data

Sruly Rosenblat, Tim O'Reilly, Ilan Strauss

机构 * AI Disclosures Project, Code for Science and Society（AI披露项目，代码用于社会） ； O’Reilly Media（O’Reilly媒体） ； Institute for Innovation and Public Purpose, University College London（创新与公共目的研究所，伦敦大学学院） 纠错

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Supervised Pretraining for Material Property Prediction

Chowdhury Mohammad Abid Rahman, Aldo H. Romero, Prashnna K. Gyawali

机构 * Lane Department of Computer Science and Electrical Engineering, West Virginia University（兰恩计算机科学与电气工程系，西弗吉尼亚大学） ； Department of Physics and Astronomy, West Virginia University（物理与天文学系，西弗吉尼亚大学） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；foundation model(abstract)；分类 cs.AI、cs.LG

Comments 21 pages, 7 figures, 2 algorithms, 6 tables

DP2FL: Dual Prompt Personalized Federated Learning in Foundation Models

Ying Chang, Xiaohu Shi, Xiaohui Zhao, Zhaohuang Chen, Deyin Ma

机构 * College of Software, Jilin University（吉林大学软件学院） ； College of Computer Science and Technology, Jilin University（吉林大学计算机科学与技术学院） ； College of Computer Science and Engineering, Changchun University of Technology（长春理工大学计算机科学与工程学院） 纠错

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

DataComp-LM: In search of the next generation of training sets for language models

Jeffrey Li, Alex Fang, Georgios Smyrnis, Maor Ivgi, Matt Jordan, Samir Gadre, Hritik Bansal, Etash Guha, Sedrick Keh, Kushal Arora, Saurabh Garg, Rui Xin, Niklas Muennighoff, Reinhard Heckel, Jean Mercat, Mayee Chen, Suchin Gururangan, Mitchell Wortsman, Alon Albalak, Yonatan Bitton, Marianna Nezhurina, Amro Abbas, Cheng-Yu Hsieh, Dhruba Ghosh, Josh Gardner, Maciej Kilian, Hanlin Zhang, Rulin Shao, Sarah Pratt, Sunny Sanyal, Gabriel Ilharco, Giannis Daras, Kalyani Marathe, Aaron Gokaslan, Jieyu Zhang, Khyathi Chandu, Thao Nguyen, Igor Vasiljevic, Sham Kakade, Shuran Song, Sujay Sanghavi, Fartash Faghri, Sewoong Oh, Luke Zettlemoyer, Kyle Lo, Alaaeldin El-Nouby, Hadi Pouransari, Alexander Toshev, Stephanie Wang, Dirk Groeneveld, Luca Soldaini, Pang Wei Koh, Jenia Jitsev, Thomas Kollar, Alexandros G. Dimakis, Yair Carmon, Achal Dave, Ludwig Schmidt, Vaishaal Shankar

机构 * University of Washington（华盛顿大学） ； Apple（苹果公司） ； Toyota Research Institute（丰田研究院） ； UT Austin（得克萨斯大学） ； Tel Aviv University（特拉维夫大学） ； Columbia University（哥伦比亚大学） ； Stanford（斯坦福） ； UCLA（加州大学洛杉矶分校） ； JSC ； LAION ； AI2 ； TUM（慕尼黑技术大学） ； CMU（卡内基梅隆大学） ； Hebrew University（希伯来大学） ； SambaNova ； Cornell（康奈尔大学） ； USC（南加州大学） ； Harvard（哈佛大学） ； UCSB（加州大学圣塔芭芭拉分校） ； SynthLabs ； Contextual AI ； DatologyAI 纠错

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Project page: https://www.datacomp.ai/dclm/

No Need to Talk: Asynchronous Mixture of Language Models

Anastasiia Filippova, Angelos Katharopoulos, David Grangier, Ronan Collobert

专题命中 预训练与数据 ：language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.LG

Comments 23 pages

From Tokens to Lattices: Emergent Lattice Structures in Language Models

Bo Xiong, Steffen Staab

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2025

Fast Adaptation with Behavioral Foundation Models

Harshit Sikchi, Andrea Tirinzoni, Ahmed Touati, Yingchen Xu, Anssi Kanervisto, Scott Niekum, Amy Zhang, Alessandro Lazaric, Matteo Pirotta

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments 25 pages

Hide and Seek in Noise Labels: Noise-Robust Collaborative Active Learning with LLM-Powered Assistance

Bo Yuan, Yulin Chen, Yin Zhang, Wei Jiang

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

VERA: Explainable Video Anomaly Detection via Verbalized Learning of Vision-Language Models

Muchao Ye, Weiyang Liu, Pan He

专题命中 预训练与数据 ：language model(title,abstract)；instruction tuning(abstract)；分类 cs.AI、cs.LG

Comments Accepted in CVPR 2025

Overcoming Vocabulary Mismatch: Vocabulary-agnostic Teacher Guided Language Modeling

Haebin Shin, Lei Ji, Xiao Liu, Yeyun Gong

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Not All LLM-Generated Data Are Equal: Rethinking Data Weighting in Text Classification

Hsun-Yu Kuo, Yin-Hsiang Liao, Yu-Chieh Chao, Wei-Yun Ma, Pu-Jen Cheng

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2025 camera ready

Specialized Foundation Models Struggle to Beat Supervised Baselines

Zongzhe Xu, Ritvik Gupta, Wenduo Cheng, Alexander Shen, Junhong Shen, Ameet Talwalkar, Mikhail Khodak

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments The first two authors contributed equally. The order was determined by coin flip

Command R7B Arabic: A Small, Enterprise Focused, Multilingual, and Culturally Aware Arabic LLM

Yazeed Alnumay, Alexandre Barbet, Anna Bialas, William Darling, Shaan Desai, Joan Devassy, Kyle Duffy, Stephanie Howe, Olivia Lasche, Justin Lee, Anirudh Shrinivason, Jennifer Tracey

专题命中 预训练与数据 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

The Power of LLM-Generated Synthetic Data for Stance Detection in Online Political Discussions

Stefan Sylvius Wagner, Maike Behrendt, Marc Ziegele, Stefan Harmeling

专题命中 预训练与数据 ：LLM(title,abstract)；prompting(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2025 Spotlight

Task-Adaptive Pretrained Language Models via Clustered-Importance Sampling

David Grangier, Simin Fan, Skyler Seto, Pierre Ablin

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 23 pages, presented at the International Conference on Learning Representation (ICLR), 2025

Hints-In-Browser: Benchmarking Language Models for Programming Feedback Generation

Nachiket Kotalwar, Alkis Gotovos, Adish Singla

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.AI、cs.LG

Scaling LLM Pre-training with Vocabulary Curriculum

Fangyuan Yu

专题命中 预训练与数据 ：LLM(title)；language model(abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

Comments under review

Krutrim LLM: Multilingual Foundational Model for over a Billion People

Aditya Kallappa, Palash Kamble, Abhinav Ravi, Akshat Patidar, Vinayak Dhruv, Deepak Kumar, Raghav Awasthi, Arveti Manjunath, Himanshu Gupta, Shubham Agarwal, Kumar Ashish, Gautam Bhargava, Chandra Khatri

专题命中 预训练与数据 ：LLM(title,abstract)；foundation model(abstract)；分类 cs.CL、cs.AI

Mitigating Data Scarcity in Time Series Analysis: A Foundation Model with Series-Symbol Data Generation

Wenxuan Wang, Kai Wu, Yujian Betterest Li, Dan Wang, Xiaoyu Zhang, Jing Liu

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG