语言大模型 / LLM - arXivDaily 专题

2203.13240 2022-03-25 cs.CL cs.LG 84%

Token Dropping for Efficient BERT Pretraining

Le Hou, Richard Yuanzhe Pang, Tianyi Zhou, Yuexin Wu, Xinying Song, Xiaodan Song, Denny Zhou

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments ACL 2022

2110.04541 2022-03-22 cs.CL cs.LG 84%

The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design

Yoav Levine, Noam Wies, Daniel Jannai, Dan Navon, Yedid Hoshen, Amnon Shashua

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2110.09456 2021-11-02 cs.CL cs.AI 84%

NormFormer: Improved Transformer Pretraining with Extra Normalization

Sam Shleifer, Jason Weston, Myle Ott

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

2102.04130 2021-10-29 cs.CL cs.AI 84%

Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models

Hannah Kirk, Yennie Jun, Haider Iqbal, Elias Benussi, Filippo Volpin, Frederic A. Dreyer, Aleksandar Shtedritski, Yuki M. Asano

专题命中预训练与数据：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to NeurIPS 2021. Code and data at https://github.com/oxai/intersectional_gpt2

2110.10319 2021-10-22 cs.CL cs.CY cs.IR cs.LG 84%

LMSOC: An Approach for Socially Sensitive Pretraining

Vivek Kulkarni, Shubhanshu Mishra, Aria Haghighi

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Camera ready version. Accepted to EMNLP 2021 Findings. Code for reproducing the experiments can be found at: https://github.com/twitter-research/lmsoc

2104.07540 2021-10-05 cs.CL cs.LG 84%

Generating Datasets with Pretrained Language Models

Timo Schick, Hinrich Schütze

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted at EMNLP2021

2109.00895 2021-09-03 cs.CV cs.AI cs.CL 84%

Knowledge Perceived Multi-modal Pretraining in E-commerce

Yushan Zhu, Huaixiao Tou, Wen Zhang, Ganqiang Ye, Hui Chen, Ningyu Zhang, Huajun Chen

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ACM MM 2021

2108.02170 2021-08-05 cs.CL cs.AI 84%

Curriculum learning for language modeling

Daniel Campos

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.AI

2107.10474 2021-07-23 cs.CL cs.LG 84%

Back-Translated Task Adaptive Pretraining: Improving Accuracy and Robustness on Text Classification

Junghoon Lee, Jounghee Kim, Pilsung Kang

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

2105.13665 2021-05-31 cs.CL cs.AI 84%

Domain-Adaptive Pretraining Methods for Dialogue Understanding

Han Wu, Kun Xu, Linfeng Song, Lifeng Jin, Haisong Zhang, Linqi Song

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 6 pages, to appear in ACL2021

2105.01735 2021-05-06 cs.CL cs.LG 84%

HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish

Robert Mroczkowski, Piotr Rybak, Alina Wróblewska, Ireneusz Gawlik

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Published in Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing

2102.12982 2021-02-26 cs.CL cs.AI cs.CV 84%

A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives

Nils Rethmeier, Isabelle Augenstein

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI

2010.15980 2020-11-10 cs.CL cs.LG 84%

AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

Taylor Shin, Yasaman Razeghi, Robert L. Logan, Eric Wallace, Sameer Singh

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments v2: Fixed error in Figure 2

2006.08671 2020-06-17 cs.CL cs.LG stat.ML 84%

To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks

Sinong Wang, Madian Khabsa, Hao Ma

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Accepted in ACL2020

2005.09207 2020-05-28 cs.IR cs.CL cs.LG 84%

Table Search Using a Deep Contextualized Language Model

Zhiyu Chen, Mohamed Trabelsi, Jeff Heflin, Yinan Xu, Brian D. Davison

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments Accepted at SIGIR 2020 (Long)

2005.00318 2020-05-04 cs.CL cs.LG 84%

Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi

Benjamin Muller, Benoit Sagot, Djamé Seddah

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

2002.01685 2020-02-06 cs.CL cs.LG 84%

Parsing as Pretraining

David Vilares, Michalina Strzyz, Anders Søgaard, Carlos Gómez-Rodríguez

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments AAAI 2020 - The Thirty-Fourth AAAI Conference on Artificial Intelligence

1906.08237 2020-01-03 cs.CL cs.LG 84%

XLNet: Generalized Autoregressive Pretraining for Language Understanding

Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, Quoc V. Le

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Pretrained models and code are available at https://github.com/zihangdai/xlnet

1912.01580 2019-12-18 cs.LG cs.CL stat.ML 84%

A Comparative Study of Pretrained Language Models on Thai Social Text Categorization

Thanapapas Horsuwan, Kasidis Kanwatchara, Peerapon Vateekul, Boonserm Kijsirikul

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments 12 pages, conference

1902.10547 2019-06-03 cs.CL cs.LG 84%

An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

Alexandra Chronopoulou, Christos Baziotis, Alexandros Potamianos

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL、cs.LG

Comments NAACL 2019

1611.02683 2018-02-23 cs.CL cs.LG cs.NE 84%

Unsupervised Pretraining for Sequence to Sequence Learning

Prajit Ramachandran, Peter J. Liu, Quoc V. Le

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments Updated to accepted EMNLP 2017 version

2608.17957 2026-08-19 cs.LG 新提交 84%

Understanding the Surprising Generalization Properties of Tabular Foundation Models

表格基础模型的出人意料的泛化特性研究

Nour Shaheen, Junwei Ma, Alex Labach, Frank Hutter, Valentin Thomas, Anthony L. Caterini

机构 * Polytechnique Montréal（蒙特利尔理工学院）； Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）； Chandar Research Lab（钱达尔研究实验室）； University of Toronto（多伦多大学）； Layer 6 AI（第六层人工智能公司）； Prior Labs（普里奥实验室）； ELLIS Institute Tübingen（埃利斯研究所图宾根分部）； University of Freiburg（弗赖堡大学）； Cohere（科here公司）

专题命中预训练与数据：foundation model(title,abstract)；分类 cs.LG

AI总结该研究揭示仅在单个真实表格上预训练的TFMs有强迁移性，提出以任务为中心、基于检索的新视角，为TFMs的模型与语料库设计提供了新框架。

Comments This work extends our previous work, Generalization Can Emerge in Tabular Foundation Models From a Single Table (arXiv:2511.09665)

2607.29129 2026-08-03 cs.LG 新提交 84%

PluRel-to-RDB-PFN: Schema-Guided Synthetic Relational Pretraining

PluRel-to-RDB-PFN：模式引导的合成关系预训练

Mohammad Sadeq Abolhasani, Viswanath Ganapathy

专题命中预训练与数据：pretraining(title,abstract)；foundation model(abstract,comments)；分类 cs.LG

AI总结该研究将合成关系数据库生成器PluRel作为RDB-PFN的外部预训练数据源，通过三种课程策略对比，发现模式优先引导策略仅用少量数据即可恢复RDB-PFN近94%的性能，证实早期接触真实模式的有效性。

Comments Proceedings of the 2nd ICML on Foundation Models for Structured Data

2606.16899 2026-06-16 cs.LG 新提交 84%

Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

奇妙预训练优化器及其发现之处 II：超球优化

Kaiyue Wen, Xingyu Dang, Kaifeng Lyu, Tengyu Ma, Percy Liang

机构 * Stanford University（斯坦福大学）； Princeton University（普林斯顿大学）； Tsinghua University（清华大学）

专题命中预训练与数据：pretraining(title,abstract)；language model(abstract)；分类 cs.LG

AI总结针对Muon等优化器在大模型预训练中增益随规模增大而减弱的问题，提出Hyperball包装器，固定权重矩阵及其更新的Frobenius范数，在1.2B参数模型上实现20-30%的token等效加速，并改善学习率迁移。

Comments Corresponding blog post: https://psychedelic-sunstone-851.notion.site/Fantastic-Pretraining-Optimizers-and-Where-to-Find-Them-2-1-Hyperball-Optimization-2e924306e6f280e7a5ffee00eb40a0dd

2412.07303 2025-04-29 cs.CL 84%

Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia

Lance Calvin Lim Gamboa, Mark Lee

机构 * School of Computer Science, University of Birmingham（英国伯明翰大学计算机科学学院）； Department of Information Systems and Computer Science, Ateneo de Manila University（马尼拉亚托大学信息系统与计算机科学系）

专题命中预训练与数据：language model(title,abstract)；pretraining(abstract)；分类 cs.CL

Comments Accepted for presentation at The First Workshop on Language Models for Low-Resource Languages (LoResLM) at The 31st International Conference on Computational Linguistics (COLING 2025)

Journal ref https://aclanthology.org/2025.loreslm-1.9/

2304.04688 2023-09-21 cs.CV cs.AI 84%

Interaction-Aware Prompting for Zero-Shot Spatio-Temporal Action Detection

Wei-Jhe Huang, Jheng-Hsien Yeh, Min-Hung Chen, Gueter Josmy Faure, Shang-Hong Lai

专题命中预训练与数据：prompting(title,abstract)；language model(abstract)；分类 cs.AI；foundation model(comments)

Comments Accepted by ICCV Workshop 2023 (What is Next in Multimodal Foundation Models?)

2607.00208 2026-07-02 cs.CL cs.AI cs.LG 新提交 83%

SLIM-RL: Risk-Budgeted Random-Masking RL for Diffusion LLMs Without Trajectory Slicing

SLIM-RL: 无需轨迹切分的扩散LLM风险预算随机掩码强化学习

Ruikang Zhao, Zhenting Wang, Han Gao, Ligong Han

机构 * Technical University of Denmark（丹麦技术大学）； MBZUAI Institute of Foundation Models（穆罕默德·本·扎耶德人工智能大学基础模型研究所）； Iowa State University（爱荷华州立大学）； Red Hat AI Innovation（红帽人工智能创新实验室）； MIT–IBM Watson AI Lab（麻省理工学院-IBM沃森人工智能实验室）

专题命中预训练与数据：LLM(title_cn)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结提出SLIM-RL方法，通过τ预算解码器控制每步提交风险，结合无迹随机掩码目标和自适应方差缩减技术，在不重构训练轨迹的情况下匹配或超越现有轨迹感知方法，在数学和代码任务上取得显著提升。

Comments 17 pages

2606.03283 2026-06-30 eess.AS cs.SD 83%

SpeakerCard-1M: An Evidence-Grounded Corpus for In-the-Wild Speaker Verification

SpeakerCard-1M：面向野外说话人确认的基于证据的说话人卡片语料库

Junyi Peng, Oldřich Plchot, Xiao Song, Dading Chong, Lichun Fan, Hang Su, Themos Stafylakis, Junjie Li, Kong Aik Lee, Shuai Wang, Jian Luan, Jan Černocký

机构 * Brno University of Technology, Czechia（布拉格技术大学，捷克）； Peking University, China（北京大学，中国）； Xiaomi, China（小米，中国）； Athens University of Economics and Business, Greece（雅典经济与商业大学，希腊）； The Hong Kong Polytechnic University, Hong Kong（香港理工大学，香港）； Nanjing University, China（南京大学，中国）

专题命中预训练与数据：LLM(summary_cn,abstract)；language model(abstract)

AI总结提出SpeakerCard-1M双语说话人资源，通过声学探针和受限LLM生成结构化说话人卡片，并定义跨模态协议以支持基于证据的说话人确认。

Comments Corpus and protocols at https://junyipeng00.github.io/SpeakerCard-1M-page

2606.27379 2026-06-29 cs.CL cs.AI cs.LG 新提交 83%

Position: The Term "Machine Unlearning" Is Overused in LLMs

立场：术语“机器遗忘”在大型语言模型中被过度使用

Sangyeon Yoon, Yeachan Jun, Albert No

专题命中预训练与数据：LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本文主张机器遗忘应限于数据集定义的删除，而许多标为“遗忘”的任务（如拒绝有害请求、实体/知识移除）实为不同目标，需不同术语和基线，并指出术语混淆导致指标误用。

Comments 13 pages; ICML 2026 Position Paper Track. Sangyeon Yoon and Yeachan Jun contributed equally

2606.21292 2026-06-23 cs.CV 新提交 83%

Lightweight 3D Feature Pretraining by Bayesian Inversion of 2D Foundation Models

轻量级3D特征预训练：基于2D基础模型的贝叶斯反演

Marwane Hariat, Gianni Franchi, David Filliat, Antoine Manzanera

机构 * U2IS, ENSTA – Institut Polytechnique de Paris, Palaiseau, France（U2IS，ENSTA – 巴黎综合理工学院，法国帕莱索）； Pôle Recherche, Agence Ministérielle pour l’IA de Défense, Palaiseau, France（研究部，国防人工智能部级机构，法国帕莱索）

专题命中预训练与数据：foundation model(title)；pretraining(title)

AI总结提出Casper3D，一种轻量级概率框架，通过贝叶斯反演将多视图2D基础模型嵌入转换为潜在3D语义表示，实现开放词汇3D理解。

1. 预训练与数据 12393 篇

Token Dropping for Efficient BERT Pretraining

The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Design

NormFormer: Improved Transformer Pretraining with Extra Normalization

Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Models

LMSOC: An Approach for Socially Sensitive Pretraining

Generating Datasets with Pretrained Language Models

Knowledge Perceived Multi-modal Pretraining in E-commerce

Curriculum learning for language modeling

Back-Translated Task Adaptive Pretraining: Improving Accuracy and Robustness on Text Classification

Domain-Adaptive Pretraining Methods for Dialogue Understanding

HerBERT: Efficiently Pretrained Transformer-based Language Model for Polish

A Primer on Contrastive Pretraining in Language Processing: Methods, Lessons Learned and Perspectives

AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts

To Pretrain or Not to Pretrain: Examining the Benefits of Pretraining on Resource Rich Tasks

Table Search Using a Deep Contextualized Language Model

Can Multilingual Language Models Transfer to an Unseen Dialect? A Case Study on North African Arabizi

Parsing as Pretraining

XLNet: Generalized Autoregressive Pretraining for Language Understanding

A Comparative Study of Pretrained Language Models on Thai Social Text Categorization

An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models

Unsupervised Pretraining for Sequence to Sequence Learning

Understanding the Surprising Generalization Properties of Tabular Foundation Models

PluRel-to-RDB-PFN: Schema-Guided Synthetic Relational Pretraining

Fantastic Pretraining Optimizers and Where to Find Them II: Hyperball Optimization

Filipino Benchmarks for Measuring Sexist and Homophobic Bias in Multilingual Language Models from Southeast Asia

Interaction-Aware Prompting for Zero-Shot Spatio-Temporal Action Detection

SLIM-RL: Risk-Budgeted Random-Masking RL for Diffusion LLMs Without Trajectory Slicing

SpeakerCard-1M: An Evidence-Grounded Corpus for In-the-Wild Speaker Verification

Position: The Term "Machine Unlearning" Is Overused in LLMs

Lightweight 3D Feature Pretraining by Bayesian Inversion of 2D Foundation Models