语言大模型 / LLM

Assessing Privacy Risks in Language Models: A Case Study on Summarization Tasks

Ruixiang Tang, Gord Lueck, Rodolfo Quispe, Huseyin A Inan, Janardhan Kulkarni, Xia Hu

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Document-Level Language Models for Machine Translation

Frithjof Petrick, Christian Herold, Pavel Petrushkov, Shahram Khadivi, Hermann Ney

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments accepted at WMT 2023

Explore, Establish, Exploit: Red Teaming Language Models from Scratch

Stephen Casper, Jason Lin, Joe Kwon, Gatlen Culp, Dylan Hadfield-Menell

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

COSA: Concatenated Sample Pretrained Vision-Language Foundation Model

Sihan Chen, Xingjian He, Handong Li, Xiaojie Jin, Jiashi Feng, Jing Liu

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI、cs.LG

Grounding Language Models to Images for Multimodal Inputs and Outputs

Jing Yu Koh, Ruslan Salakhutdinov, Daniel Fried

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published in ICML 2023. Project page: https://jykoh.com/fromage

Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications

Han Xie, Da Zheng, Jun Ma, Houyu Zhang, Vassilis N. Ioannidis, Xiang Song, Qing Ping, Sheng Wang, Carl Yang, Yi Xu, Belinda Zeng, Trishul Chilimbi

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments To be published in the KDD 2023 proceedings as a full paper

Language Modelling with Pixels

Phillip Rust, Jonas F. Lotz, Emanuele Bugliarello, Elizabeth Salesky, Miryam de Lhoneux, Desmond Elliott

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2023

AfroLM: A Self-Active Learning-based Multilingual Pretrained Language Model for 23 African Languages

Bonaventure F. P. Dossou, Atnafu Lambebo Tonja, Oreen Yousuf, Salomey Osei, Abigail Oppong, Iyanuoluwa Shode, Oluwabusayo Olufunke Awoyomi, Chris Chinenye Emezue

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Third Workshop on Simple and Efficient Natural Language Processing, EMNLP 2022

LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting

Xiang Chen, Lei Li, Shumin Deng, Chuanqi Tan, Changliang Xu, Fei Huang, Luo Si, Huajun Chen, Ningyu Zhang

专题命中 预训练与数据 ：prompting(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by COLING 2022

Adapting a Language Model While Preserving its General Knowledge

Zixuan Ke, Yijia Shao, Haowei Lin, Hu Xu, Lei Shu, Bing Liu

专题命中 预训练与数据 ：language model(title,abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments EMNLP 2022

OmniVL:One Foundation Model for Image-Language and Video-Language Tasks

Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, Lu Yuan

专题命中 预训练与数据 ：foundation model(title,abstract)；language model(abstract)；pretraining(abstract)

Comments To appear at NeurIPs 2022, Camera Ready with Typos fixed

Are Larger Pretrained Language Models Uniformly Better? Comparing Performance at the Instance Level

Ruiqi Zhong, Dhruba Ghosh, Dan Klein, Jacob Steinhardt

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ACL 2021 Findings. Code and data: https://github.com/ruiqi-zhong/acl2021-instance-level

A Mathematical Exploration of Why Language Models Help Solve Downstream Tasks

Nikunj Saunshi, Sadhika Malladi, Sanjeev Arora

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments This version is the camera-ready version for ICLR 2021. Main changes include a detailed discussion about natural tasks, more detailed proof sketch and updated experimental evaluations

Self-Alignment Pretraining for Biomedical Entity Representations

Fangyu Liu, Ehsan Shareghi, Zaiqiao Meng, Marco Basaldella, Nigel Collier

专题命中 预训练与数据 ：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NAACL 2021 camera-ready version

Investigating Transferability in Pretrained Language Models

Alex Tamkin, Trisha Singh, Davide Giovanardi, Noah Goodman

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Findings of EMNLP 2020

Training Fair Tabular Foundation Models

训练公平表格基础模型

Patrik Kenfack, Jesse C. Cresswell, Anthony L. Caterini, Samira Ebrahimi Kahou, Ulrich Aïvodji

机构 * ÉTS Montréal（蒙特利尔高等技术学院） ； Mila - Quebec AI Institute（米拉-魁北克人工智能研究所） ； Layer 6 AI（第六层人工智能公司） ； University of Calgary（卡尔加里大学） ； CIFAR（加拿大高级研究所） 纠错

专题命中 预训练与数据 ：foundation model(title,abstract)；分类 cs.AI、cs.LG

AI总结 针对表格基础模型（TFMs）公平性未被充分探索的问题，本研究提出FairTFM训练策略，将公平约束融入TFMs训练，在132个公平任务上实现公平性提升且保持竞争力准确性。

Comments Spotlight paper at the ICML 2026 Workshop on Foundation Models for Structured Data

NumLeak: Public Numeric Benchmarks as Latent Labels in Foundation Models

NumLeak: 基础模型中的公开数值基准作为潜在标签

Anany Kotawala

机构 * Princeton University（普林斯顿大学） 纠错

专题命中 预训练与数据 ：foundation model(title,comments)；LLM(abstract_cn)；pretraining(abstract)；分类 cs.AI、cs.LG

AI总结 提出NumLeak框架，通过API边界探测和开源因果模型的白盒验证，揭示基础模型在预训练中记忆公开数值基准，导致评估高估泛化能力。

Comments 23 pages, 12 figures, 17 tables. Accepted at the ICML 2026 Workshop on the Impact of Memorization on Trustworthy Foundation Models (MemFM)

Toward Cross-Lingual Quality Classifiers for Multilingual Pretraining Data Selection

迈向多语言预训练数据选择的跨语言质量分类器

Yassine Turki, Vinko Sabolčec, Bettina Messmer, Martin Jaggi

机构 * Machine Learning Optimization Lab（机器学习优化实验室） ； Ecole Polytechnique Fédérale de Lausanne（洛桑联邦理工学院） 纠错

专题命中 预训练与数据 ：pretraining(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

AI总结 本文探讨了通过跨语言迁移、第三四分位采样和保留率调整策略，提升多语言数据筛选效果，证明大规模多语言池化在稳定性与准确率上优于单语基线，尤其对低资源语言有显著提升。

Comments Accepted at the 3rd Workshop on Navigating and Addressing Data Problems for Foundation Models (DATA-FM @ ICLR 2026). 31 pages, 4 figures

Leveraging Generic Time Series Foundation Models for EEG Classification

Théo Gnassounou, Yessin Moakher, Shifeng Xie, Vasilii Feofanov, Ievgen Redko

机构 * Université Paris-Saclay, Inria, CEA（巴黎萨克雷大学、法国国家信息与自动化研究所、法国原子能委员会） ； Huawei Noah’s Ark Lab（华为诺亚实验室） 纠错

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Journal ref NeurIPS 2025 Workshop "Recent Advances in Time Series Foundation Models Have We Reached the 'BERT Moment'?"

Corrigibility as a Singular Target: A Vision for Inherently Reliable Foundation Models

Ram Potham, Max Harms

机构 * Independent Researcher（独立研究者） ； Machine Intelligence Research Institute（机器智能研究院） 纠错

专题命中 预训练与数据 ：foundation model(title,abstract)；SFT(abstract)；分类 cs.AI、cs.LG

Comments Preprint. This work has been submitted to the Reliable and Responsible Foundation Models Workshop at ICML 2025 for review

Sailor: Open Language Models for South-East Asia

Longxu Dou, Qian Liu, Guangtao Zeng, Jia Guo, Jiahui Zhou, Wei Lu, Min Lin

专题命中 预训练与数据 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI；LLM(comments)

Comments Code is available at https://github.com/sail-sg/sailor-llm

Foundation Models for Semantic Novelty in Reinforcement Learning

Tarun Gupta, Peter Karkus, Tong Che, Danfei Xu, Marco Pavone

专题命中 预训练与数据 ：foundation model(title,abstract)；pretraining(abstract)；分类 cs.AI、cs.LG

Comments Foundation Models for Decision Making Workshop at Neural Information Processing Systems, 2022

Modeling Protein Using Large-scale Pretrain Language Model

Yijia Xiao, Jiezhong Qiu, Ziang Li, Chang-Yu Hsieh, Jie Tang

专题命中 预训练与数据 ：language model(title,abstract)；pretraining(abstract,comments)；分类 cs.CL、cs.LG

Comments Accepted paper in Pretrain@KDD 2021 (The International Workshop on Pretraining: Algorithms, Architectures, and Applications)

EntropyMoE: Entropy-Aware Sparse Expert Routing for Tokenizer-Free LLMs

EntropyMoE：面向无分词器大语言模型的熵感知稀疏专家路由

Bo Liu, Muxuab Yu, Yu Zhang, Pengfei Gao, Yongping Zhang

专题命中 预训练与数据 ：LLM(summary_cn,abstract_cn)；large language model(abstract)；language model(abstract)；分类 cs.AI

AI总结 EntropyMoE针对无分词器字节级LLM的块计算局限性，提出基于块熵的稀疏专家路由MoE架构，在降低位每字节的同时保持下游准确率，扩展了MoE建模的应用范围。

The Tamed Subgradient Unadjusted Langevin Algorithm beyond Convexity

超越凸性的驯服次梯度未校正朗之万算法

Iosif Lytras, Nikolaos Makras, Sotirios Sabanis

机构 * University of Edinburgh（爱丁堡大学） ； National Technical University of Athens（雅典国立技术大学） ； Athena/Archimedes Research Centre（雅典娜/阿基米德研究中心） 纠错

专题命中 预训练与数据 ：LLM(summary_cn,abstract)；pretraining(abstract)；分类 cs.LG

AI总结 本文针对非光滑、超线性梯度增长且非凸的目标分布采样问题，提出SG-TULA算法，推导其非渐近收敛界，验证假设并用于GPT-2系列LLM预训练，效果优于AdamW等。

Comments 53 pages

Cochise: A Reference Harness for Autonomous Penetration Testing

Cochise：自主渗透测试的参考框架

Andreas Happe, Jürgen Cito

机构 * TU Wien（维也纳技术大学） 纠错

专题命中 预训练与数据 ：LLM(summary_cn,abstract)；prompting(abstract)；分类 cs.AI

AI总结 Cochise提供了一个简洁的自主渗透测试框架，支持通过SSH连接LLM代理与Linux主机，并通过分离的规划-执行架构实现可控环境，同时提供重放和分析工具以支持研究。

DuplexGen: Adaptive Synthesis of Human-AI Turn-Taking Dialogues

DuplexGen：人机交替对话的自适应合成

Takyoung Kim, Kang-wook Kim, Sang Hoon Woo, Julia Hirschberg, Gunhee Kim, Dilek Hakkani-Tür

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； Seoul National University（首尔大学） ； Columbia University（哥伦比亚大学） ； University of California, Berkeley（加州大学伯克利分校） ； Georgia Institute of Technology（佐治亚理工学院） 纠错

专题命中 预训练与数据 ：LLM(summary_cn,abstract)；prompting(abstract)；分类 cs.CL

AI总结 DuplexGen框架通过将LLM预测与少量槽级人类偏好标注校准，生成场景自适应交替发言对话，契合人类偏好的效果优于未校准方法，证明人类校准对交替发言合成场景特定性的关键作用。

Comments Manuscript under review

Design-Specification Tiling for ICL-based CAD Code Generation

基于ICL的CAD代码生成的Design-Specification Tiling设计

Yali Du, San-Zhuo Xi, Hui Sun, Ming Li

机构 * National Key Laboratory for Novel Software Technology, School of Artificial Intelligence, Nanjing University（新型软件技术国家实验室，人工智能学院，南京大学） 纠错

专题命中 预训练与数据 ：large language model(abstract,abstract_cn)；language model(abstract,abstract_cn)；prompting(abstract)；分类 cs.LG

AI总结 本文提出DST方法，通过量化知识充分性提升CAD代码生成质量，优于现有ICL示例选择策略。

Heuresis: Search Strategies for Autonomous AI Research Agents Across Quality, Diversity and Novelty

Heuresis: 自主AI研究智能体在质量、多样性和新颖性上的搜索策略

Antonis Antoniades, Deepak Nathani, Ritam Saha, Alfonso Amayuelas, Ivan Bercovich, Zhaotian Weng, Vignesh Baskaran, Kunal Bhatia, William Yang Wang

机构 * University of California, Santa Barbara（加州大学圣塔芭芭拉分校） ； Hexo AI 纠错

专题命中 预训练与数据 ：LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)；pretraining(abstract)

AI总结 提出Heuresis框架，将研究流程抽象为通用原语，实现开放科学探索；在三个领域评估六种搜索策略，发现完全新颖的想法罕见且无法达到最高性能，揭示了当前策略无法扩展质量-新颖性前沿的挑战。

Comments 14 pages main text, 82 pages total including appendix; 38 figures, 4 tables

MGUP: A Momentum-Gradient Alignment Update Policy for Stochastic Optimization

MGUP：一种用于随机优化的动量-梯度对齐更新策略

Da Chang, Ganzhao Yuan

机构 * Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究院） ； Shenzhen University of Advanced Technology（深圳理工大学） ； Pengcheng Laboratory（鹏城实验室） ； University of Chinese Academy of Sciences（中国科学院大学） 纠错

专题命中 预训练与数据 ：LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)；pretraining(abstract)

AI总结 提出MGUP机制，通过按固定比例选择参数施加大步长、其余参数用小步长，增强动量优化器，理论保证收敛，实验表明提升训练效率与稳定性。

Comments Published in NeurIPS 2025