语言大模型 / LLM - arXivDaily 专题

2411.12843 2024-11-21 cs.LG cs.AI cs.CL stat.ML 75%

Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

Shang Liu, Yu Pan, Guanting Chen, Xiaocheng Li

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2409.09774 2024-11-07 cs.CV 75%

Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through $f$-divergence Minimization

Haoyuan Sun, Bo Xia, Yongzhe Chang, Xueqian Wang

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments 34 pages

2405.19534 2024-11-01 cs.LG cs.AI cs.CL 75%

Preference Learning Algorithms Do Not Learn Preference Rankings

Angelica Chen, Sadhika Malladi, Lily H. Zhang, Xinyi Chen, Qiuyi Zhang, Rajesh Ranganath, Kyunghyun Cho

专题命中后训练与偏好优化：LLM(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NeurIPS 2024 camera-ready

2401.12086 2024-10-28 cs.CL cs.AI cs.LG 75%

West-of-N: Synthetic Preferences for Self-Improving Reward Models

Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, Aliaksei Severyn

专题命中后训练与偏好优化：language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2410.11677 2024-10-21 cs.CL cs.AI cs.LG 75%

Understanding Likelihood Over-optimisation in Direct Alignment Algorithms

Zhengyan Shi, Sander Land, Acyr Locatelli, Matthieu Geist, Max Bartolo

专题命中后训练与偏好优化：language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint Version

2410.03717 2024-10-08 cs.CL cs.AI cs.LG 75%

Revisiting the Superficial Alignment Hypothesis

Mohit Raghavendra, Vaskar Nath, Sean Hendryx

专题命中后训练与偏好优化：language model(abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

2403.10704 2024-09-16 cs.LG cs.AI cs.CL 75%

Parameter Efficient Reinforcement Learning from Human Feedback

Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, Bowen Li, Saravanan Ganesh, Bill Byrne, Jessica Hoffmann, Hassan Mansoor, Wei Li, Abhinav Rastogi, Lucas Dixon

专题命中后训练与偏好优化：language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2404.10501 2024-08-22 cs.CV cs.AI cs.CL cs.LG 75%

Self-Supervised Visual Preference Alignment

Ke Zhu, Zheng Ge, Liang Zhao, Xiangyu Zhang

专题命中后训练与偏好优化：language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments MM2024 oral

2408.10270 2024-08-21 cs.LG cs.AI cs.CL 75%

SEAL: Systematic Error Analysis for Value ALignment

Manon Revel, Matteo Cargnelutti, Tyna Eloundou, Greg Leppert

专题命中后训练与偏好优化：language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 28 pages, 17 Figures, 8 Tables

2408.10075 2024-08-20 cs.LG cs.AI cs.CL cs.RO 75%

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, Natasha Jaques

专题命中后训练与偏好优化：foundation model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments weirdlabuw.github.io/vpl

2405.18952 2024-06-04 cs.CL cs.AI cs.LG 75%

Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets

Peter Devine

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2312.10240 2024-04-10 cs.CV 75%

Rich Human Feedback for Text-to-Image Generation

Youwei Liang, Junfeng He, Gang Li, Peizhao Li, Arseniy Klimovskiy, Nicholas Carolan, Jiao Sun, Jordi Pont-Tuset, Sarah Young, Feng Yang, Junjie Ke, Krishnamurthy Dj Dvijotham, Katie Collins, Yiwen Luo, Yang Li, Kai J Kohlhoff, Deepak Ramachandran, Vidhya Navalpakkam

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments CVPR'24

2310.13595 2023-11-29 cs.CY 75%

The History and Risks of Reinforcement Learning and Human Feedback

Nathan Lambert, Thomas Krendl Gilbert, Tom Zick

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments 14 pages, 3 figures

2307.04349 2023-11-14 cs.AI cs.CL cs.LG 75%

RLTF: Reinforcement Learning from Unit Test Feedback

Jiate Liu, Yiqin Zhu, Kaiwen Xiao, Qiang Fu, Xiao Han, Wei Yang, Deheng Ye

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by TMLR

2310.19080 2023-11-07 cs.CV 75%

Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery

Katie Z Luo, Zhenzhen Liu, Xiangyu Chen, Yurong You, Sagie Benaim, Cheng Perng Phoo, Mark Campbell, Wen Sun, Bharath Hariharan, Kilian Q. Weinberger

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)

2308.12030 2023-10-03 cs.CL cs.AI cs.LG 75%

Prompt-Based Length Controlled Generation with Reinforcement Learning

Renlong Jie, Xiaojun Meng, Lifeng Shang, Xin Jiang, Qun Liu

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2304.07297 2023-06-13 cs.AI cs.CL cs.LG cs.MA 75%

Language Instructed Reinforcement Learning for Human-AI Coordination

Hengyuan Hu, Dorsa Sadigh

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2208.11981 2023-05-10 cs.CL cs.AI cs.LG 75%

On Reality and the Limits of Language Data: Aligning LLMs with Human Norms

Nigel H. Collier, Fangyu Liu, Ehsan Shareghi

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 9 pages; data available, see https://sites.google.com/site/nhcollier/projects/art

2301.06859 2023-01-18 cs.HC cs.AI cs.CL cs.LG 75%

Methodological reflections for AI alignment research using human feedback

Thilo Hagendorff, Sarah Fabi

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2411.06424 2025-06-10 cs.LG cs.CL 74%

How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis

Yushi Yang, Filip Sondej, Harry Mayne, Andrew Lee, Adam Mahdi

机构 * University of Oxford（牛津大学）； Jagiellonian University（雅盖隆大学）； Harvard University（哈佛大学）

专题命中后训练与偏好优化：language model(abstract,journal_ref)；preference optimization(abstract)；分类 cs.CL、cs.LG

Journal ref NeurIPS 2024 Workshop on Socially Responsible Language Modelling Research (SoLaR)

2204.05862 2022-04-13 cs.CL cs.LG 74%

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, Jared Kaplan

专题命中后训练与偏好优化：RLHF(abstract,comments)；language model(abstract)；分类 cs.CL、cs.LG

Comments Data available at https://github.com/anthropics/hh-rlhf

2607.28829 2026-08-03 cs.NI cs.LG 新提交 74%

When Unlearning Fails: Reliable Data Deletion under Post-Training in Agent Networks

当遗忘失效时：智能体网络后训练下的可靠数据删除

Zihao Ding, Jun Huang, Liang Dong

专题命中后训练与偏好优化：post-training(title)；分类 cs.LG

AI总结针对自改进联邦智能体网络后训练时数据删除易出现影响回声的问题，提出MUTE方法，经实验验证其可在保障任务效用的同时降低行为泄漏且通信开销更低。

2605.17497 2026-05-19 cs.LG 74%

Self-Supervised On-Policy Distillation for Reasoning Language Models

自监督在线策略蒸馏用于推理语言模型

Zhiquan Tan, Yinrong Hong

机构 * Tsinghua University（清华大学）； Beihang University（北航大学）

专题命中后训练与偏好优化：language model(title)；分类 cs.LG

AI总结本文提出自监督在线策略蒸馏（SSOPD）方法，通过对比正确与错误的完成过程信号，提升推理语言模型的表现，实验表明在多个基准测试中优于GRPO和OPSD基线。

2604.00200 2026-04-02 cs.LG 74%

Offline Constrained RLHF with Multiple Preference Oracles

离线约束强化学习中的多偏好Oracle

Brenden Latham, Mehrdad Moharrami

机构 * University of Iowa（爱荷华大学）

专题命中后训练与偏好优化：RLHF(title)；分类 cs.LG

AI总结本文基于多偏好Oracle研究离线约束强化学习，通过最大似然估计Oracle奖励并分析统计不确定性传播，将约束目标转化为KL-正则化拉格朗日函数，提出仅需对偶的算法以保证约束满足并提供有限样本性能保证。

2603.23550 2026-03-26 cs.LG 74%

Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction

隐式回合式策略优化用于主动用户-大语言模型交互

Haoyu Wang, Yuxin Chen, Liang Luo, Buyun Zhang, Ellie Dingqiao Wen, Pan Li

机构 * Georgia Institute of Technology（佐治亚理工学院）； Meta AI

专题命中后训练与偏好优化：LLM(title)；分类 cs.LG

AI总结本文提出隐式回合式策略优化方法，通过隐式过程奖励模型从稀疏结果信号中提取细粒度回合级奖励，提升多轮协作任务的训练稳定性与收敛性。

2603.16065 2026-03-24 cs.RO cs.AI 74%

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models

大奖励模型：基于视觉-语言模型的通用在线机器人奖励生成

Yanru Wu, Weiduo Yuan, Ang Qi, Vitor Guizilini, Jiageng Mao, Yue Wang

机构 * USC Physical Superintelligence Lab（USC物理超智能实验室）； Toyota Research Institute（丰田研究院）

专题命中后训练与偏好优化：language model(title)；分类 cs.AI

AI总结本文提出利用视觉-语言模型生成通用在线机器人奖励，通过零样本方式提升策略学习效率，实验显示在30次迭代内显著提高初始策略成功率。

2603.16978 2026-03-19 cs.RO cs.LG 74%

Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models

奖励DINO：基于视觉基础模型预测密集奖励

Pierre Krack, Tobias Jülg, Wolfram Burgard, Florian Walter

机构 * Department of Computer Science & Artificial Intelligence, University of Technology Nuremberg（技术大学纽伦堡计算机科学与人工智能系）； TUM School of Computation, Information and Technology, Technical University of Munich（慕尼黑技术大学计算、信息与技术学院）

专题命中后训练与偏好优化：foundation model(title)；分类 cs.LG

AI总结本文提出Rewarding DINO，一种基于语言条件的奖励模型，通过学习实际奖励函数而非特定轨迹，实现机器人操作任务的密集奖励预测，具有紧凑结构和低计算开销，能有效泛化至新场景。

Comments 10 pages, 5 figures, submitted to IEEE

2512.04552 2026-02-17 cs.SD cs.AI eess.AS 74%

RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS

RRPO: 为基于LLM的情感TTS的鲁棒奖励策略优化

Cong Wang, Changfeng Gao, Yang Xiang, Zhihao Du, Keyu An, Han Zhao, Qian Chen, Xiangang Li, Yingming Gao, Ya Li

机构 * Speech Team, Tongyi Lab, Alibaba Group（通义实验室，阿里集团）

专题命中后训练与偏好优化：LLM(title)；分类 cs.AI

AI总结 RRPO通过鲁棒奖励模型和混合正则化方案，提升LLM情感TTS的鲁棒性和自然度，有效对抗奖励黑客攻击。

Comments Accepted by ICASSP 2026. Copyright 2026 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

2510.27410 2026-02-02 cs.AI 74%

Closing the Expression Gap in LLM Instructions via Socratic Questioning

通过苏格拉底提问缩小LLM指令的表达差距

Jianwen Sun, Yukang Feng, Yifan Chang, Chuanhao Li, Zizhen Li, Jiaxin Ai, Fanrui Zhang, Yu Dai, Kaipeng Zhang

机构 * Nankai University（南开大学）； Shanghai Innovation Institute（上海创新研究院）； Wuhan University（武汉大学）； Shanghai AI Laboratory（上海人工智能实验室）； Shanda AI Research（上海沙达人AI研究所）

专题命中后训练与偏好优化：LLM(title)；分类 cs.AI

AI总结通过苏格拉底提问方法，提出Nous代理以主动探测用户意图，解决人类与AI协作中的意图表达差距问题，提升效率和输出质量。

2504.01223 2025-10-31 cs.LG math.PR 74%

Explainable post-training bias mitigation with distribution-based fairness metrics

Ryan Franks, Alexey Miroshnikov, Konstandinos Kotsiopoulos

机构 * Discover Financial Services（Discover金融服务公司）

专题命中后训练与偏好优化：post-training(title)；分类 cs.LG

Comments 45 pages, 4 figures

1. 后训练与偏好优化 4535 篇

Reward Modeling with Ordinal Feedback: Wisdom of the Crowd

Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through $f$-divergence Minimization

Preference Learning Algorithms Do Not Learn Preference Rankings

West-of-N: Synthetic Preferences for Self-Improving Reward Models

Understanding Likelihood Over-optimisation in Direct Alignment Algorithms

Revisiting the Superficial Alignment Hypothesis

Parameter Efficient Reinforcement Learning from Human Feedback

Self-Supervised Visual Preference Alignment

SEAL: Systematic Error Analysis for Value ALignment

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

Are You Sure? Rank Them Again: Repeated Ranking For Better Preference Datasets

Rich Human Feedback for Text-to-Image Generation

The History and Risks of Reinforcement Learning and Human Feedback

RLTF: Reinforcement Learning from Unit Test Feedback

Reward Finetuning for Faster and More Accurate Unsupervised Object Discovery

Prompt-Based Length Controlled Generation with Reinforcement Learning

Language Instructed Reinforcement Learning for Human-AI Coordination

On Reality and the Limits of Language Data: Aligning LLMs with Human Norms

Methodological reflections for AI alignment research using human feedback

How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

When Unlearning Fails: Reliable Data Deletion under Post-Training in Agent Networks

Self-Supervised On-Policy Distillation for Reasoning Language Models

Offline Constrained RLHF with Multiple Preference Oracles

Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interaction

Large Reward Models: Generalizable Online Robot Reward Generation with Vision-Language Models

Rewarding DINO: Predicting Dense Rewards with Vision Foundation Models

RRPO: Robust Reward Policy Optimization for LLM-based Emotional TTS

Closing the Expression Gap in LLM Instructions via Socratic Questioning

Explainable post-training bias mitigation with distribution-based fairness metrics