语言大模型 / LLM

Scalable Vision-Language-Action Model Pretraining for Robotic Manipulation with Real-Life Human Activity Videos

Qixiu Li, Yu Deng, Yaobo Liang, Lin Luo, Lei Zhou, Chengtang Yao, Lingqi Zeng, Zhiyuan Feng, Huizhi Liang, Sicheng Xu, Yizhong Zhang, Xi Chen, Hao Chen, Lily Sun, Dong Chen, Jiaolong Yang, Baining Guo

机构 * Tsinghua University（清华大学） ； Microsoft Research Asia（微软亚洲研究院） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Comments Project page: https://microsoft.github.io/VITRA/

ViTime: Foundation Model for Time Series Forecasting Powered by Vision Intelligence

Luoxiao Yang, Yun Wang, Xinqi Fan, Israel Cohen, Jingdong Chen, Zijun Zhang

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

Journal ref Transactions on Machine Learning Research,2025

Born a Transformer -- Always a Transformer? On the Effect of Pretraining on Architectural Abilities

Mayank Jobanputra, Yana Veitsman, Yash Sarrof, Aleksandra Bakalova, Vera Demberg, Ellie Pavlick, Michael Hahn

机构 * Saarland University（萨尔兰大学） ； Brown University（布朗大学） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2025

Diffusion Transformers as Open-World Spatiotemporal Foundation Models

Yuan Yuan, Chonghua Han, Jingtao Ding, Guozhen Zhang, Depeng Jin, Yong Li

机构 * Department of Electronic Engineering, BNRist, Tsinghua University（电子工程系，BNRist，清华大学） 纠错

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted by NeurIPS 2025

Discursive Circuits: How Do Language Models Understand Discourse Relations?

Yisong Miao, Min-Yen Kan

机构 * Web IR / NLP Group (WING)（网络信息检索与自然语言处理组） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Comments Accepted to EMNLP 2025 (Main Conference); 9 pages, 8 figures, 5 tables (20 pages, 12 figures, 14 tables including references and appendices)

BLAZER: Bootstrapping LLM-based Manipulation Agents with Zero-Shot Data Generation

Rocktim Jyoti Das, Harsh Singh, Diana Turmakhan, Muhammad Abdullah Sohail, Mingfei Han, Preslav Nakov, Fabio Pizzati, Ivan Laptev

机构 * Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；分类 cs.AI、cs.LG

Comments 11 pages, 8 figures

Meta-Pretraining for Zero-Shot Cross-Lingual Named Entity Recognition in Low-Resource Philippine Languages

David Demitri Africa, Suchir Salhan, Yuval Weiss, Paula Buttery, Richard Diehl Martinez

机构 * University of Cambridge（剑桥大学） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted (poster) to 5th Workshop on Multilingual Representation Learning at EMNLP 2025

Point-RTD: Replaced Token Denoising for Pretraining Transformer Models on Point Clouds

Gunner Stone, Youngsook Choi, Alireza Tavakkoli, Ankita Shukla

机构 * Department of Computer Science ； Engineering, University of Nevada, Reno, USA 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

A Simple Review of EEG Foundation Models: Datasets, Advancements and Future Perspectives

Junhong Lai, Jiyu Wei, Lin Yao, Yueming Wang

机构 * The College of Computer Science, Zhejiang University, Hangzhou, China.（浙江大学计算机科学学院） ； The MOE Frontiers Science Center for Brain and Brain-Machine Integration, Zhejiang University, Hangzhou, China.（脑科学与脑机接口前沿科学中心） ； The Nanhu Brain-Computer Interface Institute, Hangzhou, China.（南湖脑机接口研究院） ； The Qiushi Academy for Advanced Studies, Hangzhou, China.（启石先进研究院） 纠错

专题命中 预训练与数据 ：foundation model(title)；pretraining(abstract)；分类 cs.AI、cs.LG

SABR: A Stable Adaptive Bitrate Framework Using Behavior Cloning Pretraining and Reinforcement Learning Fine-Tuning

Pengcheng Luo, Yunyang Zhao, Bowen Zhang, Genke Yang, Boon-Hee Soong, Chau Yuen

机构 * Ningbo Artificial Intelligence Institute, Shanghai Jiao Tong University, Ningbo, China（宁波人工智能研究院，上海交通大学，宁波，中国） ； School of Automation and Intelligent Sensing, Shanghai Jiao Tong University, Shanghai, China（自动化与智能感知学院，上海交通大学，上海，中国） ； School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore（电子与电气工程学院，南洋理工大学，新加坡） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Slaves to the Law of Large Numbers: An Asymptotic Equipartition Property for Perplexity in Generative Language Models

Tyler Bell, Avinash Mudireddy, Ivan Johnson-Eversoll, Soura Dasgupta, Raghu Mudumbai

机构 * University of Iowa（爱荷华大学） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

CEHR-XGPT: A Scalable Multi-Task Foundation Model for Electronic Health Records

Chao Pang, Jiheum Park, Xinzhuo Jiang, Nishanth Parameshwar Pavinkurve, Krishna S. Kalluri, Shalmali Joshi, Noémie Elhadad, Karthik Natarajan

机构 * Columbia University Irving Medical Center（哥伦比亚大学伊万格琳医疗中心） ； Observational Health Data Sciences and Informatics（观察性健康数据科学与信息学） ； Department of Biomedical Informatics（生物医学信息学系） ； Columbia University（哥伦比亚大学） ； NewYork-Presbyterian Hospital（纽约- Presbyterian医院） ； Department of Computer Science（计算机科学系） 纠错

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

Multitask Battery Management with Flexible Pretraining

Hong Lu, Jiali Chen, Jingzhao Zhang, Guannan He, Xuebing Han, Minggao Ouyang

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

FlexOlmo: Open Language Models for Flexible Data Use

Weijia Shi, Akshita Bhagia, Kevin Farhat, Niklas Muennighoff, Pete Walsh, Jacob Morrison, Dustin Schwenk, Shayne Longpre, Jake Poznanski, Allyson Ettinger, Daogao Liu, Margaret Li, Dirk Groeneveld, Mike Lewis, Wen-tau Yih, Luca Soldaini, Kyle Lo, Noah A. Smith, Luke Zettlemoyer, Pang Wei Koh, Hannaneh Hajishirzi, Ali Farhadi, Sewon Min

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

PaPaformer: Language Model from Pre-trained Parallel Paths

Joonas Tapaninaho, Mourad Oussala

机构 * University of Oulu, Faculty of Information Technology and Electrical Engineering, CMVS（奥卢大学，信息科技与电气工程学院） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

H-Net++: Hierarchical Dynamic Chunking for Tokenizer-Free Language Modelling in Morphologically-Rich Languages

Mehrdad Zakershahrak, Samira Ghodratnama

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

A Scalable Pretraining Framework for Link Prediction with Efficient Adaptation

Yu Song, Zhigang Hua, Harry Shomer, Yan Xie, Jingzhe Liu, Bo Long, Hui Liu

机构 * Computer Science and Engineering（计算机科学与工程） ； Michigan State University（密歇根州立大学） ； Meta 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted by KDD 2025 Research Track

VAULT: Vigilant Adversarial Updates via LLM-Driven Retrieval-Augmented Generation for NLI

Roie Kazoom, Ofir Cohen, Rami Puzis, Asaf Shabtai, Ofer Hadar

专题命中 预训练与数据 ：LLM(title,abstract)；分类 cs.AI、cs.LG

Robust Classification under Noisy Labels: A Geometry-Aware Reliability Framework for Foundation Models

Ecem Bozkurt, Antonio Ortega

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

Comments 5 pages, 2 figures, under review at CAMSAP 2025

Uncovering Gradient Inversion Risks in Practical Language Model Training

Xinguo Feng, Zhongkui Ma, Zihan Wang, Eu Joe Chegne, Mengyao Ma, Alsharif Abuadbba, Guangdong Bai

机构 * The University of Queensland（昆士兰大学） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.AI、cs.LG

Comments 15 Pages, 5 figures, 10 tables. Accepted by ACM CCS 2024

Journal ref ACM CCM 2024

When Does Metadata Conditioning (NOT) Work for Language Model Pre-Training? A Study with Context-Free Grammars

Rei Higuchi, Ryotaro Kawata, Naoki Nishikawa, Kazusato Oko, Shoichiro Yamaguchi, Sosuke Kobayashi, Seiya Tokui, Kohei Hayashi, Daisuke Okanohara, Taiji Suzuki

机构 * The University of Tokyo（东京大学） ； RIKEN AIP（日本研究机构（Advanced Institute for Proprietary Technology）） ； University of California, Berkeley（加州大学伯克利分校） ； Preferred Networks, Inc.（Preferred Networks公司） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.LG

Language model developers should report train-test overlap

Andy K Zhang, Kevin Klyman, Yifan Mai, Yoav Levine, Yian Zhang, Rishi Bommasani, Percy Liang

机构 * Stanford University（斯坦福大学） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.AI、cs.LG

Comments ICML 2025 Spotlight; 23 pages

Training Dynamics Underlying Language Model Scaling Laws: Loss Deceleration and Zero-Sum Learning

Andrei Mircea, Supriyo Chakraborty, Nima Chitsazan, Milind Naphade, Sambit Sahu, Irina Rish, Ekaterina Lobacheva

机构 * Mila – Quebec AI Institute（魁北克人工智能研究所） ； Université de Montréal（蒙特利尔大学） ； Capital One 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.AI、cs.LG

Comments Published as a conference paper at ACL 2025

Style over Substance: Distilled Language Models Reason Via Stylistic Replication

Philip Lippmann, Jie Yang

机构 * Delft University of Technology（代尔夫特理工大学） 纠错

专题命中 预训练与数据 ：language model(title,abstract)；分类 cs.CL、cs.AI

Comments To appear at COLM 2025

Referential ambiguity and clarification requests: comparing human and LLM behaviour

Chris Madge, Matthew Purver, Massimo Poesio

机构 * Queen Mary University of London（伦敦大学Queen Mary） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；分类 cs.CL、cs.AI

Next-token pretraining implies in-context learning

Paul M. Riechers, Henry R. Bigelow, Eric A. Alt, Adam Shai

机构 * Simplex（简单公司） ； Astera Institute（Astera研究所） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Human2LocoMan: Learning Versatile Quadrupedal Manipulation with Human Pretraining

Yaru Niu, Yunzhe Zhang, Mingyang Yu, Changyi Lin, Chenhao Li, Yikai Wang, Yuxiang Yang, Wenhao Yu, Tingnan Zhang, Zhenzhen Li, Jonathan Francis, Bingqing Chen, Jie Tan, Ding Zhao

机构 * Carnegie Mellon University（卡内基梅隆大学） ； Google DeepMind（谷歌DeepMind） ； Bosch Center for AI（博世人工智能中心） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG

Real-TabPFN: Improving Tabular Foundation Models via Continued Pre-training With Real-World Data

Anurag Garg, Muhammad Ali, Noah Hollmann, Lennart Purucker, Samuel Müller, Frank Hutter

机构 * Department of Informatik, University of Freiburg（弗赖堡大学信息学院） ； Prior Labs GmbH ； Meta ； ELLIS Institute Tübingen（图宾根ELLIS研究所） 纠错

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

FairCauseSyn: Towards Causally Fair LLM-Augmented Synthetic Data Generation

Nitish Nagesh, Ziyu Wang, Amir M. Rahmani

机构 * University of California, Irvine（加州大学尔湾分校） 纠错

专题命中 预训练与数据 ：LLM(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted to IEEE EMBC 2025

Bootstrapping your behavior: a new pretraining strategy for user behavior sequence data

Weichang Wu, Xiaolu Zhang, Jun Zhou, Yuchen Li, Wenwen Xia

机构 * Ant Group（蚂蚁集团） ； Singapore Management University（新加坡管理学院） ； Soochow University（苏州大学） 纠错

专题命中 预训练与数据 ：pretraining(title,abstract)；分类 cs.AI、cs.LG