语言大模型 / LLM - arXivDaily 专题

2405.19107 2024-05-30 cs.LG cs.AI 90%

Offline Regularised Reinforcement Learning for Large Language Models Alignment

Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, Bilal Piot

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.AI、cs.LG

2405.00578 2024-05-02 cs.CL cs.AI 90%

The Real, the Better: Aligning Large Language Models with Online Human Behaviors

Guanying Jiang, Lingyong Yan, Haibo Shi, Dawei Yin

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI

Comments 11 pages, 6 figures

2404.01342 2024-04-03 cs.CL cs.AI 90%

DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model

Lirui Zhao, Yue Yang, Kaipeng Zhang, Wenqi Shao, Yuxin Zhang, Yu Qiao, Ping Luo, Rongrong Ji

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI

Comments Published as a conference paper at CVPR 2024

2403.14409 2024-03-22 cs.CL cs.AI 90%

Locating and Mitigating Gender Bias in Large Language Models

Yuchen Cai, Ding Cao, Rongxi Guo, Yaqin Wen, Guiquan Liu, Enhong Chen

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI

Comments 23 pages, 5 figures

2402.03659 2024-03-01 cs.LG cs.CL q-fin.ST 90%

Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models

Kelvin J. L. Koa, Yunshan Ma, Ritchie Ng, Tat-Seng Chua

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.LG

Comments WWW 2024

2402.14245 2024-02-23 cs.RO cs.AI cs.LG 90%

Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models

Jinyi Liu, Yifu Yuan, Jianye Hao, Fei Ni, Lingzhi Fu, Yibin Chen, Yan Zheng

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.AI、cs.LG

Comments Presented at AAAI 2024 RL+LLMs Workshop

2401.15024 2024-02-12 cs.LG cs.CL 90%

SliceGPT: Compress Large Language Models by Deleting Rows and Columns

Saleh Ashkboos, Maximilian L. Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, James Hensman

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；post-training(abstract)；分类 cs.CL、cs.LG

Comments 22 pages, 8 figures, accepted at ICLR24

2401.12292 2024-02-01 cs.CL cs.AI 90%

GRATH: Gradual Self-Truthifying for Large Language Models

Weixin Chen, Dawn Song, Bo Li

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

2401.02870 2024-01-08 cs.MA cs.AI cs.CL 90%

AFSPP: Agent Framework for Shaping Preference and Personality with Large Language Models

Zihong He, Changwang Zhang

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI

2311.08487 2023-11-16 cs.CL cs.AI 90%

Alignment is not sufficient to prevent large language models from generating harmful information: A psychoanalytic perspective

Zi Yin, Wei Ding, Jia Liu

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；post-training(abstract)；分类 cs.CL、cs.AI

2310.10076 2023-10-17 cs.CL cs.AI 90%

Verbosity Bias in Preference Labeling by Large Language Models

Keita Saito, Akifumi Wachi, Koki Wataoka, Youhei Akimoto

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

2502.10391 2025-02-17 cs.CL cs.CV 90%

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

Yi-Fan Zhang, Tao Yu, Haochen Tian, Chaoyou Fu, Peiyan Li, Jianshu Zeng, Wulin Xie, Yang Shi, Huanyu Zhang, Junkang Wu, Xue Wang, Yibo Hu, Bin Wen, Fan Yang, Zhang Zhang, Tingting Gao, Di Zhang, Liang Wang, Rong Jin, Tieniu Tan

专题命中后训练与偏好优化：RLHF(title,abstract)；LLM(title)；large language model(abstract)；language model(abstract)

Comments Project Page: https://mm-rlhf.github.io/

2608.12158 2026-08-13 cs.CV 新提交 89%

Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization

DPO中的上下文盲区：通过上下文校准的偏好优化缓解多模态大语言模型（MLLMs）中的物体幻觉

Byungoh Ko, Jinyoung Park, Jongha Kim, Jeehye Na, Jaewon Cho, Hyunwoo J. Kim

机构 * Korea University（高丽大学）； KAIST（韩国科学技术院）

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract,abstract_cn)；language model(abstract,abstract_cn)

AI总结本研究针对MLLMs的物体幻觉问题，提出C²-DPO方法，通过最大化上下文偏好增益降低幻觉率，使Qwen2-VL-Instruct-2B的幻觉率相对降低36%且不损害通用推理能力。

Comments Accepted at ECCV2026

2608.06789 2026-08-10 cs.NI 新提交 89%

EvoRIC: Reinforcement Learning Fine-Tuned LLM-empowered RAN Intelligent Control Toward Autonomous O-RAN

EvoRIC：面向自主O-RAN的、由强化学习微调大语言模型赋能的RAN智能控制器

Lingyan Bao, Jemin Lee, Tony Q. S. Quek

专题命中后训练与偏好优化：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)

AI总结针对传统RAN智能算法泛化差、通用LLM算力高且缺领域知识的问题，提出EvoRIC分层框架，结合RLFT与PPO优化LLM，在IAB网络中验证其泛化性与效能，为自主O-RAN提供新方案。

Comments Manuscript submitted 23 April 2026; revised 7 August 2026

2607.03528 2026-07-07 cs.LG cs.AI cs.CL 新提交 89%

Aligning Language Models with Selective Prediction

通过选择性预测使语言模型对齐

Gaoxiang Luo, Yifan Wu, Sinian Zhang, Aryan Deshwal, Ju Sun

机构 * Department of Computer Science and Engineering（计算机科学与工程系）； Bioinformatics and Computational Biology Program（生物信息学与计算生物学项目）； Division of Biostatistics and Health Data Science（生物统计学与健康数据科学部）； University of Minnesota Twin Cities（明尼苏达大学双城分校）

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；post-training(abstract)

AI总结研究聚焦提升语言模型可靠性，采用选择性预测策略，在模型训练后对齐阶段，提出基于强化学习的框架RLSR，以风险-覆盖曲线下面积为目标，在域内域外任务中风险-覆盖权衡表现更好。

Comments Accepted by ICML 2026 Agents in the Wild: Safety, Security, and Beyond Workshop, Project Page: https://sun-umn.github.io/RLSR

2606.25509 2026-06-25 cs.RO cs.CV 新提交 89%

ASSCG: Just-Right Gating over Chattering for Fast-Slow LLM Planning in Autonomous Driving

ASSCG：自动驾驶中快慢LLM规划的恰到好处门控

Sining Ang, Yuan Chen, Liu Haiyan, Xuanyao Mao, Jason Bao, Xuliang, Bingchuan Sun, Yan Wang

机构 * Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院）； Department of Automation, University of Science and Technology of China（中国科学技术大学自动化系）； Beijing University of Aeronautics and Astronautics（北京航空航天大学）； Lenovo Group Limited（联想集团）

专题命中后训练与偏好优化：LLM(title,title_cn)；large language model(abstract)；language model(abstract)

AI总结提出自适应慢系统控制门（ASSCG），通过帧级查询/缓存/丢弃决策优化快慢规划器调用，结合RWKV骨干网络和GRPO风格强化学习，在nuPlan和NAVSIM上分别提升性能并降低延迟。

2410.15595 2026-06-18 cs.AI cs.CL cs.LG 版本更新 89%

A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications

直接偏好优化综述：数据集、理论、变体及应用

Wenyi Xiao, Zechuan Wang, Leilei Gan, Shuai Zhao, Zongrui Li, Ruirui Lei, Wanggui He, Luu Anh Tuan, Long Chen, Hao Jiang, Zhou Zhao, Fei Wu

机构 * Zhejiang University（浙江大学）； Nanyang Technological University（南洋理工大学）； Alibaba Group（阿里巴巴集团）

专题命中后训练与偏好优化：preference optimization(title,abstract)；RLHF(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结综述直接偏好优化（DPO）在理论、变体、数据集和应用方面的进展，指出其作为RL-free替代方案的潜力与局限，并提出未来研究方向。

Comments Accepted by TPAMI 2026. Project page: https://github.com/Mr-Loevan/DPO-Survey

2604.03238 2026-06-02 cs.HC 89%

RLHF May Not Reflect Genuine Preferences

RLHF 可能不反映真实偏好

Bijean Ghafouri, Eun Cheol Choi, Priyanka Dey, Emilio Ferrara

专题命中后训练与偏好优化：RLHF(title,title_cn)

AI总结本文指出 RLHF 中的标注响应可能并非真实偏好，通过提出测量有效性诊断方法，发现不一致性具有系统性偏差，过滤高不一致标注者会显著改变模型输出。

2605.24597 2026-05-26 cs.AI cs.CL cs.LG 89%

Learning to Reason Efficiently with A* Post-Training

学习通过A*后训练进行高效推理

Andreas Opedal, Francesco Ignazio Re, Abulhair Saparov, Mrinmaya Sachan, Bernhard Schölkopf, Ryan Cotterell

机构 * ETH Zürich（苏黎世联邦理工学院）； MPI for Intelligent Systems, Tübingen（图宾根智能系统研究所）； Purdue University（普渡大学）

专题命中后训练与偏好优化：LLM(summary_cn,abstract_cn)；post-training(title)；large language model(abstract)；language model(abstract)

AI总结本文通过A*搜索算法指导LLM生成正确且高效的推理步骤，提出监督微调和强化学习两种训练方法，在1B-3B参数模型上显著提升推理准确性和效率。

Comments Preprint

2605.16517 2026-05-21 cs.SE 89%

Customizing an LLM for Enterprise Software Engineering

为企业软件工程定制LLM

Aditya Kini, Satish Chandra, Milad Hashemi, Saksham Thakur, Aditya Pandey, Vincent Nguyen, Marc Brockschmidt, Franjo Ivančić, Danny Tarlow, Parthasarathy Ranganathan, Petros Maniatis, Ahmed Omran, Zaheer Abbas, Anita Gergely, Martin Sevenich, Gufeng Zhang, Amy Hua, Alexander Frömmgen

专题命中后训练与偏好优化：LLM(title,title_cn)；post-training(abstract)

AI总结本文提出Gemini for Google，一种专为企业内部软件工程生态系统定制的LLM，通过端到端开发和中间训练策略，显著提升了开发效率和代码存活率，为其他组织提供了可复制的数据利用路径。

Comments 11 pages, 8 figures

2605.12112 2026-05-13 cs.CV 89%

When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy

当策略熵约束失效时：通过感知熵在基于流的RLHF中保持多样性

Xiaofeng Tan, Jun Liu, Bin-Bin Gao, Yuanting Fan, Xi Jiang, Chengjie Wang, Hongsong Wang, Feng Zheng

机构 * Southeast University（东南大学）； Tencent Youtu Lab（腾讯云图实验室）； Southern University of Science and Technology（南方科技大学）

专题命中后训练与偏好优化：RLHF(title,title_cn)

AI总结本文研究了基于流的RLHF中策略熵与多样性之间的关系，提出感知熵作为新的约束方法，通过在感知空间中保持多样性提升模型质量。

2504.10766 2026-05-12 cs.LG cs.AI cs.CL 89%

How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients

指令和推理数据如何塑造训练后阶段：通过层梯度的视角探讨数据质量

Ming Li, Yanhong Li, Ziyue Li, Tianyi Zhou

机构 * University of Maryland（马里兰大学）； Allen Institute for AI（Allen人工智能研究所）

专题命中后训练与偏好优化：post-training(title,abstract)；LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结本文通过层梯度的谱分析，揭示了高质量指令和推理数据对大语言模型训练后阶段的影响，统一了数据评估指标，并展示了数据质量与训练稳定性之间的关系。

Comments ACL2026, camera-ready

2602.20532 2026-02-25 cs.LG cs.AI cs.CL 89%

Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training

Actor-Curator: 通过策略改进带状机为RL后训练实现联合适应课程学习

Zhengyao Gu, Jonathan Light, Raul Astudillo, Ziyu Ye, Langzhou He, Henry Peng Zou, Wei Cheng, Santiago Paternain, Philip S. Yu, Yisong Yue

机构 * University of Illinois Chicago（伊利诺伊大学香槟分校）； Caltech（加州理工学院）； RPI（罗切斯特理工学院）； MBZUAI（澳门大学人工智能研究院）； University of Chicago（芝加哥大学）； NEC Laboratories America（NEC美国实验室）

专题命中后训练与偏好优化：post-training(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

AI总结 ACTOR-CURATOR通过策略改进带状机实现RL后训练的联合适应课程学习，有效提升训练稳定性和效率。

Comments 37 pages, 8 figures, 1 table. Preprint under review. Equal contribution by first two authors

2510.08256 2025-10-10 cs.LG cs.AI cs.CL 89%

Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization

Jason Bohne, Pawel Polak, David Rosenberg, Brian Bloniarz, Gary Kazantsev

机构 * Department of Applied Mathematics and Statistics（应用数学与统计学系）； Stony Brook University（石溪大学）； Institute for Advanced Computational Science（先进计算科学研究所）； Center of Excellence in Wireless and Information Technology (CEWIT)（无线与信息技术卓越中心）； AI Innovation Institute（人工智能创新研究院）； Bloomberg（彭博）； Toronto, ON M5J 2S1（多伦多，ON M5J 2S1）； San Francisco, CA 94105（旧金山，CA 94105）； Bloomberg New York, NY 10022（纽约，NY 10022）

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

2506.08022 2025-10-10 cs.LG cs.AI cs.CL cs.CV 89%

Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining

Chenxi Liu, Tianyi Xiong, Yanshuo Chen, Ruibo Chen, Yihan Wu, Junfeng Guo, Tianyi Zhou, Heng Huang

机构 * University of Maryland, College Park（马里兰大学学院 park）

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

2503.02832 2025-07-24 cs.CL cs.AI cs.LG 89%

AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation

Songming Zhang, Xue Zhang, Tong Zhang, Bojie Hu, Yufeng Chen, Jinan Xu

机构 * Key Laboratory of Big Data & Artificial Intelligence in Transportation, (Beijing Jiaotong University), Ministry of Education（大数据与人工智能交通运输联合实验室，（北京交通大学）教育部）； School of Computer Science and Technology, Beijing Jiaotong University, Beijing, China（计算机科学与技术学院，北京交通大学，北京，中国）； Tencent Inc, China（腾讯公司，中国）

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract)；large language model(abstract)；RLHF(abstract)

Comments ACL 2025 Main Conference, code available at: https://github.com/songmzhang/AlignDistil

2410.10148 2025-07-22 cs.LG cs.AI cs.CL 89%

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization

Junkang Wu, Xue Wang, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, Xiangnan He

机构 * MoE Key Lab of BIPC, University of Science（BIPC联合实验室，科学与技术大学）； Alibaba Group（阿里巴巴集团）

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

2407.10490 2025-07-01 cs.LG cs.AI cs.CL 89%

Learning Dynamics of LLM Finetuning

Yi Ren, Danica J. Sutherland

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；instruction tuning(abstract)

2411.07595 2025-06-16 cs.LG cs.AI cs.CL 89%

Entropy Controllable Direct Preference Optimization

Motoki Omura, Yasuhiro Fujita, Toshiki Kataoka

机构 * The University of Tokyo（东京大学）

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；post-training(abstract)

Comments ICML 2025 Workshop on Models of Human Feedback for AI Alignment

2410.02197 2025-06-12 cs.AI cs.CL cs.LG 89%

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment

Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, Quanquan Gu

机构 * IIIS, Tsinghua University, Beijing, China（清华大学信息科学技术学院，北京，中国）； Shanghai Qi Zhi Institute, Shanghai, China（上海启智研究院，上海，中国）； Department of Computer Science, University of California, Los Angeles, California, USA（计算机科学系，加州大学洛杉矶分校，美国，加州）

专题命中后训练与偏好优化：language model(title,abstract)；foundation model(abstract)；post-training(abstract)；RLHF(abstract)

Comments Accepted to the 42nd International Conference on Machine Learning (ICML 2025)

1. 后训练与偏好优化 4507 篇

Offline Regularised Reinforcement Learning for Large Language Models Alignment

The Real, the Better: Aligning Large Language Models with Online Human Behaviors

DiffAgent: Fast and Accurate Text-to-Image API Selection with Large Language Model

Locating and Mitigating Gender Bias in Large Language Models

Learning to Generate Explainable Stock Predictions using Self-Reflective Large Language Models

Enhancing Robotic Manipulation with AI Feedback from Multimodal Large Language Models

SliceGPT: Compress Large Language Models by Deleting Rows and Columns

GRATH: Gradual Self-Truthifying for Large Language Models

AFSPP: Agent Framework for Shaping Preference and Personality with Large Language Models

Alignment is not sufficient to prevent large language models from generating harmful information: A psychoanalytic perspective

Verbosity Bias in Preference Labeling by Large Language Models

MM-RLHF: The Next Step Forward in Multimodal LLM Alignment

Context Blindness in DPO: Mitigating Object Hallucination in MLLMs via Context-Calibrated Preference Optimization

EvoRIC: Reinforcement Learning Fine-Tuned LLM-empowered RAN Intelligent Control Toward Autonomous O-RAN

Aligning Language Models with Selective Prediction

ASSCG: Just-Right Gating over Chattering for Fast-Slow LLM Planning in Autonomous Driving

A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applications

RLHF May Not Reflect Genuine Preferences

Learning to Reason Efficiently with A* Post-Training

Customizing an LLM for Enterprise Software Engineering

When Policy Entropy Constraint Fails: Preserving Diversity in Flow-based RLHF via Perceptual Entropy

How Instruction and Reasoning Data shape Post-Training: Data Quality through the Lens of Layer-wise Gradients

Actor-Curator: Co-adaptive Curriculum Learning via Policy-Improvement Bandits for RL Post-Training

Mix- and MoE-DPO: A Variational Inference Approach to Direct Preference Optimization

Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining

AlignDistil: Token-Level Language Model Alignment as Adaptive Policy Distillation

AlphaDPO: Adaptive Reward Margin for Direct Preference Optimization

Learning Dynamics of LLM Finetuning

Entropy Controllable Direct Preference Optimization

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment