语言大模型 / LLM - arXivDaily 专题

2305.17608 2025-10-31 cs.LG cs.AI cs.CL math.OC stat.ML 89%

Reward Collapse in Aligning Large Language Models

Ziang Song, Tianle Cai, Jason D. Lee, Weijie J. Su

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted for publication in the Journal of Data Science (JDS), reference JDS1201

2502.20548 2025-10-21 cs.LG cs.AI cs.CL 89%

$Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training

Jin Peng Zhou, Kaiwen Wang, Jonathan Chang, Zhaolin Gao, Nathan Kallus, Kilian Q. Weinberger, Kianté Brantley, Wen Sun

专题命中后训练与偏好优化：LLM(title,abstract)；post-training(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NeurIPS 2025

2506.00419 2025-09-11 cs.CR cs.SE 89%

Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences

Mohammad Saqib Hasan, Saikat Chakraborty, Santu Karmaker, Niranjan Balasubramanian

专题命中后训练与偏好优化：LLM(title,abstract)；preference optimization(title,abstract)；prompting(abstract)

Comments Accepted to ACL 2025 (Main)

2411.09073 2025-07-10 cs.CL cs.AI cs.LG 89%

CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedback

Wenbo Zhang, Aditya Majumdar, Amulya Yadav

机构 * Pennsylvania State University（宾夕法尼亚州立大学）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments full draft v2: 8 pages, 3 figures

2502.09604 2025-06-17 cs.CL cs.AI cs.LG 89%

SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models

Yung-Sung Chuang, Benjamin Cohen-Wang, Shannon Zejiang Shen, Zhaofeng Wu, Hu Xu, Xi Victoria Lin, James Glass, Shang-Wen Li, Wen-tau Yih

机构 * Massachusetts Institute of Technology（麻省理工学院）； Meta FAIR

专题命中后训练与偏好优化：large language model(title)；language model(title)；LLM(abstract)；preference optimization(abstract)

Comments ICML 2025 main conference paper. The source code is available at https://github.com/facebookresearch/SelfCite

2402.17826 2024-12-05 cs.LG cs.AI cs.CL cs.CY cs.HC stat.ML 89%

Prediction-Powered Ranking of Large Language Models

Ivi Chatzi, Eleni Straitouri, Suhas Thejaswi, Manuel Gomez Rodriguez

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published at NeurIPS 2024

2409.00787 2024-09-04 cs.CL cs.AI cs.CR cs.LG 89%

The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs

Bocheng Chen, Hanqing Guo, Guangjing Wang, Yuanda Wang, Qiben Yan

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2404.05829 2024-07-19 cs.CL cs.AI cs.LG 89%

SambaLingo: Teaching Large Language Models New Languages

Zoltan Csaki, Bo Li, Jonathan Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, Urmish Thakker

专题命中后训练与偏好优化：large language model(title)；language model(title)；LLM(abstract)；preference optimization(abstract)

Comments 23 pages

2406.11817 2024-06-18 cs.CL cs.AI cs.LG 89%

Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level

Jie Liu, Zhanhui Zhou, Jiaheng Liu, Xingyuan Bu, Chao Yang, Han-Sen Zhong, Wanli Ouyang

专题命中后训练与偏好优化：language model(title,abstract)；preference optimization(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2406.04300 2024-06-07 cs.RO 89%

Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Models

Phat Nguyen, Tsun-Hsuan Wang, Zhang-Wei Hong, Sertac Karaman, Daniela Rus

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)

Comments 14 pages, 7 figures

2404.15100 2024-04-24 cs.CV cs.MM 89%

Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation

Xun Wu, Shaohan Huang, Furu Wei

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；instruction tuning(abstract)

2403.04640 2024-03-08 cs.CV 89%

CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios

Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, Xiaochun Cao

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；preference optimization(abstract)

2304.00416 2023-04-04 cs.AI cs.CL cs.CY cs.HC cs.LG 89%

Towards Healthy AI: Large Language Models Need Therapists Too

Baihan Lin, Djallel Bouneffouf, Guillermo Cecchi, Kush R. Varshney

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2509.12991 2026-01-30 cs.LG cs.AI stat.AP 89%

Bridging Performance Gaps for ECG Foundation Models: A Post-Training Strategy

弥合ECG基础模型性能差距：一种训练后策略

Ya Zhou, Yujie Yang, Xiaohan Fan, Wei Zhao

机构 * Department of Information Center, Fuwai Hospital, Chinese Academy of Medical Sciences and Peking Union Medical College（信息中心部门，阜外医院，中国医学科学院和北京协和医学院）； Function Test Center, Fuwai Hospital, National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College（功能测试中心，阜外医院，国家心血管疾病中心，中国医学科学院和北京协和医学院）； Cardiac Arrhythmia Center, Fuwai Hospital, National Center for Cardiovascular Diseases, Chinese Academy of Medical Sciences and Peking Union Medical College（心律失常中心，阜外医院，国家心血管疾病中心，中国医学科学院和北京协和医学院）； Center for Health Statistics and Information, National Health Commission People’s Republic of China（健康统计与信息中心，中华人民共和国国家卫生健康委员会）

专题命中后训练与偏好优化：post-training(title,abstract)；foundation model(title,abstract)；分类 cs.AI、cs.LG

AI总结本文提出一种训练后策略，通过提升ECG基础模型的性能，使其在多个任务中表现优于基线微调方法。

Comments A Transformer-based model is used as an example; the proposed post-training strategy may also be applicable to CNN-based models. The manuscript is currently under review

2608.16739 2026-08-18 cs.LG 新提交 89%

Le Critique: Privileged Value Functions for LLM Reinforcement Learning

Le Critique：用于大语言模型强化学习的特权值函数

Siddarth Venkatraman, Matthieu Dinot, Laurence Aitchison

机构 * Mistral AI（米斯特拉尔人工智能公司）； Mila – Quebec AI Institute（米拉-魁北克人工智能研究所）； Université de Montréal（蒙特利尔大学）

专题命中后训练与偏好优化：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)；分类 cs.LG

AI总结该研究针对 LLM 强化学习的值函数应用难题，提出特权值函数（PVF）与自适应插值基线 TETHER，在多推理任务中提升了值函数基线性能，表现优于或媲美 GRPO。

2608.15041 2026-08-18 cs.AI 新提交 89%

LLM-Based Hierarchical Coordinated Control with Continuation-Aware Policy Learning

基于大语言模型的分层协调控制与感知延续性的策略学习

Changhong He, Jinda Gao, Xinkuan Liu, Le Zhang, Xizi Luo, Yu Mei

专题命中后训练与偏好优化：LLM(title,summary_cn)；prompting(abstract)；分类 cs.AI

AI总结针对复杂工程系统多单元协调难题，提出基于LLM的分层框架，结合感知延续性的GRPO，在多匝道交通控制和VPP能源管理中，性能优于多种对比方法。

Comments 30 pages, 5 figures

2605.03799 2026-08-14 cs.CL 版本更新 89%

Natural Language Processing: A Comprehensive Practical Guide from Tokenisation to RLHF

自然语言处理：从分词到RLHF的全面实用指南

Mullosharaf K. Arabov

机构 * KAZAN (VOLGA REGION) FEDERAL UNIVERSITY INSTITUTE OF COMPUTATIONAL MATHEMATICS AND INFORMATION TECHNOLOGIES（卡赞（伏尔加地区）联邦大学计算数学与信息科技学院）

专题命中后训练与偏好优化：RLHF(title,title_cn)；LLM(abstract_cn)；large language model(abstract)；language model(abstract)

AI总结本文系统指导现代NLP全流程，涵盖分词、向量化、大语言模型微调、检索增强生成及人类反馈强化学习，强调低资源语言处理与开源模型应用。

Comments 136 pages, 12 practical works, preprint. Textbook for senior undergraduates and graduate students. Original contributions on low-resource languages (Tajik, Tatar and other). Companion repository available

2604.18401 2026-08-12 cs.CL 版本更新 89%

CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities

StepPO: 面向智能体强化学习的步骤对齐策略优化

Daoyu Wang, Qingchuan Li, Mingyue Cheng, Jie Ouyang, Shuo Yu, Chunli Liu, Shijin Wang, Qi Liu, Enhong Chen

机构 * State Key Laboratory of Cognitive Intelligence, University of Science and Technology of China（认知智能国家重点实验室，中国科学技术大学）

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract_cn)

AI总结提出StepPO，一种步骤级策略优化方法，通过将智能体强化学习从token级MDP重构为步骤级MDP并引入步骤级信用分配，以解决现有算法在智能体决策粒度上的不匹配问题，在多跳问答等任务上优于多种RL算法。

2608.06526 2026-08-10 cs.CL 新提交 89%

GRASP: Reinforcing Language Model Anonymizers with Group Relative Policy Optimization

GRASP：用组相对策略优化增强语言模型匿名化器

Sajjad Ghiasvand, Nader Sehatbakhsh

机构 * UC Santa Barbara（加州大学圣巴巴拉分校）； UC Los Angeles（加州大学洛杉矶分校）

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；preference optimization(abstract)

AI总结该研究提出GRASP方法，用组相对策略优化增强设备端小型语言模型匿名化器，在隐私-效用权衡、对抗匿名化防御及运行成本上均优于DPO蒸馏基线。

2608.03952 2026-08-05 cs.AI 新提交 89%

TACT: Taxonomy-Aligned Post-Training for Pedagogically Adaptive English Tutoring

TACT：面向教学自适应英语辅导的分类对齐后训练

Dongjie Yang, Siyan Lin, Leixian Shen, Rui Sheng, Huamin Qu, Zixin Chen

专题命中后训练与偏好优化：post-training(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结本研究提出基于人类研究的TACT框架，构建含双分类体系的数据集后，通过特定训练得到TACTutor，在自研基准及盲法研究中表现优于基线，为自适应ESL辅导器开发提供开放基础。

2601.14032 2026-07-28 cs.CL 版本更新 89%

RM-Distiller: Exploiting Generative LLM for Reward Model Distillation

RM-Distiller: 利用生成式大语言模型进行奖励模型蒸馏

Hongli Zhou, Hui Huang, Wei Liu, Chenglong Wang, Xingyuan Bu, Lvyuan Han, Fuhai Song, Muyun Yang, Wenhao Jiang, Hailong Cao, Tiejun Zhao

机构 * Faculty of Computing, Harbin Institute of Technology（哈尔滨工业大学计算机学院）； School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）

专题命中后训练与偏好优化：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)；分类 cs.CL

AI总结 RM-Distiller通过系统利用教师LLM的精炼、评分和生成能力，提升奖励模型蒸馏效果，首次系统性地探索了生成式LLM在奖励建模中的应用。

Comments Accepted to IJCAI-ECAI 2026

2607.09796 2026-07-21 cs.LG 版本更新 89%

Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels

噪声偏好标签下无元数据的元重加权直接偏好优化

Hua Qu, Yifan Li, Xiaodong Yuan

机构 * Xi’an Jiaotong University（西安交通大学）

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结研究针对DPO性能依赖偏好数据质量问题，提出双层优化框架、无任务元知识驱动方法及结合中心差分近似与LoRA微调的可扩展训练方案，经实验验证该方法能在不同噪声率下提升训练性能。

Comments 36 pages, including appendices. Revised version with updated theoretical analysis, supplementary material, figures and improved table formatting

2606.28998 2026-07-07 cs.SE cs.AI 版本更新 89%

Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation

从预训练或微调的大语言模型进行无奖励代码对齐：揭示代码生成中的权衡

Sanjeepan Sivapiran, Gias Uddin

机构 * York University Canada（加拿大约克大学）

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

AI总结本研究通过实证分析，探讨了在代码生成任务中，对预训练或微调后的大语言模型进行无奖励对齐（DPO和BoNBoN）的效果，发现预训练到对齐的路径提升更大，但微调版本基线更高。

Journal ref The ACM International Conference on the Foundations of Software Engineering (FSE) 2026

2602.01357 2026-06-09 cs.LG 版本更新 89%

Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning

你的自对弈算法其实是一个对抗性模仿者：通过模仿学习的视角理解LLM自对弈

Shangzhe Li, Xuchao Zhang, Chetan Bansal, Weitong Zhang

机构 * University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）； Microsoft Research（微软研究院）

专题命中后训练与偏好优化：LLM(title,title_cn)；large language model(abstract)；language model(abstract)；post-training(abstract)

AI总结本文通过将自对弈微调建模为模型与自身参数化的正则化隐式奖励玩家之间的极小极大博弈，统一了自对弈模仿与偏好对齐，并提出了基于χ²散度的新算法，在多种语言模型微调任务上优于现有方法。

Comments 26 pages, 6 tables, 5 figures

2603.05881 2026-06-04 cs.CL 89%

Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimation

回答前先置信：高效LLM不确定性估计的范式转变

Changcheng Li, Jiancan Wu, Hengheng Zhang, Zhengsu Chen, Guo An, Junxiang Qiu, Xiang Wang, Qi Tian

机构 * University of Science and Technology of China（中国科学技术大学）； Huawei Inc.（华为公司）

专题命中后训练与偏好优化：LLM(title,title_cn)；large language model(abstract)；language model(abstract)；分类 cs.CL

AI总结提出CoCA框架，通过GRPO强化学习联合优化置信度校准与答案准确性，实现回答前输出置信度，提升不确定性估计效率。

2602.07639 2026-06-03 cs.CL 89%

Letting Tutor Personas Speak Up for LLMs: Learning Steering Vectors from Dialogue via Preference Optimization

让导师角色为LLMs发声：通过偏好优化从对话中学习引导向量

Jaewook Lee, Alexander Scarlatos, Simon Woodhead, Andrew Lan

机构 * University of Massachusetts Amherst（马萨诸塞大学阿姆赫斯特分校）； Eedi

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结本文提出使用偏好优化训练引导向量，从人类导师-学生对话中提取导师角色信息，以控制大语言模型的行为，实现多样化的教学风格。

Comments Accepted to ACL 2026 BEA Workshop

2603.22335 2026-05-28 cs.IR cs.AI 89%

Causal Direct Preference Optimization for Distributionally Robust Generative Recommendation

因果直接偏好优化用于分布鲁棒的生成式推荐

Chu Zhao, Enneng Yang, Jianzhe Zhao, Guibing Guo

机构 * Northeastern University, Shenyang, China（东北大学，沈阳，中国）； Shenzhen Campus of Sun Yat-sen University, China（中山大学深圳校区，中国）

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结针对直接偏好优化（DPO）在生成式推荐中放大环境混杂因素导致的虚假相关性问题，提出CausalDPO，通过因果不变性学习、后门调整和软聚类环境建模来提升分布外泛化性能。

Comments 22 pages, 3 figures

2605.21883 2026-05-27 cs.CL 89%

Token-weighted Direct Preference Optimization with Attention

基于注意力的令牌加权直接偏好优化

Chengyu Huang, Zhuohang Li, Sheng-Yen Chou, Claire Cardie

机构 * Cornell University（康奈尔大学）； Vanderbilt University（范德比大学）

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结提出Token-weighted DPO (TwDPO)方法，利用注意力机制估计令牌权重，在不增加额外训练成本的情况下提升大语言模型与人类偏好对齐的性能。

2512.15605 2026-05-26 cs.LG stat.ML 89%

Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction

自回归语言模型实际上是能量模型：对下一个词元预测的预见能力的洞察

Mathieu Blondel, Michael E. Sander, Germain Vivier-Ardisson, Tianlin Liu, Vincent Roulet

机构 * Google DeepMind（谷歌DeepMind）

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；post-training(abstract)

AI总结本文通过建立自回归模型与能量模型之间的双射，揭示了自回归模型在下一个词元预测范式下具备预见能力，并提供了理论误差界。

2505.23803 2026-05-26 cs.CR cs.AI 89%

MultiPhishGuard: An Explainable and Adaptive Multi-Agent LLM System for Phishing Email Detection

MultiPhishGuard: 一种用于钓鱼邮件检测的可解释且自适应的多智能体大语言模型系统

Yinuo Xue, Eric Spero, Meng Wai Woo, Wei Gao, Giovanni Russello

机构 * The University of Auckland（奥克兰大学）

专题命中后训练与偏好优化：LLM(title,summary_cn)；prompting(abstract)；分类 cs.AI

AI总结提出基于LLM的多智能体框架MultiPhishGuard，通过协调文本、URL、元数据等五个专业智能体并利用PPO动态加权，结合对抗训练提升对新型钓鱼策略的鲁棒性，在公开数据集上达到97.89%准确率。

1. 后训练与偏好优化 4507 篇

Reward Collapse in Aligning Large Language Models

$Q\sharp$: Provably Optimal Distributional RL for LLM Post-Training

Teaching an Old LLM Secure Coding: Localized Preference Optimization on Distilled Preferences

CHAI for LLMs: Improving Code-Mixed Translation in Large Language Models through Reinforcement Learning with AI Feedback

SelfCite: Self-Supervised Alignment for Context Attribution in Large Language Models

Prediction-Powered Ranking of Large Language Models

The Dark Side of Human Feedback: Poisoning Large Language Models via User Inputs

SambaLingo: Teaching Large Language Models New Languages

Iterative Length-Regularized Direct Preference Optimization: A Case Study on Improving 7B Language Models to GPT-4 Level

Text-to-Drive: Diverse Driving Behavior Synthesis via Large Language Models

Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generation

CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios

Towards Healthy AI: Large Language Models Need Therapists Too

Bridging Performance Gaps for ECG Foundation Models: A Post-Training Strategy

Le Critique: Privileged Value Functions for LLM Reinforcement Learning

LLM-Based Hierarchical Coordinated Control with Continuation-Aware Policy Learning

Natural Language Processing: A Comprehensive Practical Guide from Tokenisation to RLHF

CAPO: Critic-Guided Action-Aligned Policy Optimization for Advancing LLM Agent Capabilities

GRASP: Reinforcing Language Model Anonymizers with Group Relative Policy Optimization

TACT: Taxonomy-Aligned Post-Training for Pedagogically Adaptive English Tutoring

RM-Distiller: Exploiting Generative LLM for Reward Model Distillation

Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels

Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM: Unpacking the Trade-offs for Code Generation

Your Self-Play Algorithm is Secretly an Adversarial Imitator: Understanding LLM Self-Play through the Lens of Imitation Learning

Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimation

Letting Tutor Personas Speak Up for LLMs: Learning Steering Vectors from Dialogue via Preference Optimization

Causal Direct Preference Optimization for Distributionally Robust Generative Recommendation

Token-weighted Direct Preference Optimization with Attention

Autoregressive Language Models are Secretly Energy-Based Models: Insights into the Lookahead Capabilities of Next-Token Prediction

MultiPhishGuard: An Explainable and Adaptive Multi-Agent LLM System for Phishing Email Detection