语言大模型 / LLM

Adversary-Aware DPO: Enhancing Safety Alignment in Vision Language Models via Adversarial Training

Fenghua Weng, Jian Lou, Jun Feng, Minlie Huang, Wenjie Wang

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)

VidChain: Chain-of-Tasks with Metric-based Direct Preference Optimization for Dense Video Captioning

Ji Soo Lee, Jongha Kim, Jeehye Na, Jinyoung Park, Hyunwoo J. Kim

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Comments AAAI 2025

Geometric-Averaged Preference Optimization for Soft Preference Labels

Hiroki Furuta, Kuang-Huei Lee, Shixiang Shane Gu, Yutaka Matsuo, Aleksandra Faust, Heiga Zen, Izzeddin Gur

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at NeurIPS 2024

MaxMin-RLHF: Alignment with Diverse Human Preferences

Souradip Chakraborty, Jiahao Qiu, Hui Yuan, Alec Koppel, Furong Huang, Dinesh Manocha, Amrit Singh Bedi, Mengdi Wang

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

SymDPO: Boosting In-Context Learning of Large Multimodal Models with Symbol Demonstration Direct Preference Optimization

Hongrui Jia, Chaoya Jiang, Haiyang Xu, Wei Ye, Mengfan Dong, Ming Yan, Ji Zhang, Fei Huang, Shikun Zhang

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Exploring RL-based LLM Training for Formal Language Tasks with Programmed Rewards

Alexander G. Padula, Dennis J. N. J. Soemers

专题命中 后训练与偏好优化 ：LLM(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at BNAIC 2024

Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison

Judy Hanwen Shen, Archit Sharma, Jun Qin

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Working Paper

Best-of-Venom: Attacking RLHF by Injecting Poisoned Preference Data

Tim Baumgärtner, Yang Gao, Dana Alon, Donald Metzler

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

UltraFeedback: Boosting Language Models with Scaled AI Feedback

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, Zhiyuan Liu, Maosong Sun

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICML 2024 camera ready

It Takes Two: On the Seamlessness between Reward and Policy Model in RLHF

Taiming Lu, Lingfeng Shen, Xinyu Yang, Weiting Tan, Beidi Chen, Huaxiu Yao

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Decoding-time Realignment of Language Models

Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, Mathieu Blondel

专题命中 后训练与偏好优化 ：language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments In Proceedings of the 41st International Conference on Machine Learning (ICML 2024)

Dataset Reset Policy Optimization for RLHF

Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D. Lee, Wen Sun

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 28 pages, 6 tables, 3 Figures, 3 Algorithms

Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, Yuhui Shi

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)

Comments Accepted by CVPR 2024

Training Language Models with Language Feedback at Scale

Jérémy Scheurer, Jon Ander Campos, Tomasz Korbak, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, Ethan Perez

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published in TMLR: https://openreview.net/forum?id=xo3hI5MwvU

NOTE: Notable generation Of patient Text summaries through Efficient approach based on direct preference optimization

Imjin Ahn, Hansle Gwon, Young-Hak Kim, Tae Joon Jun, Sanghyun Park

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Comments 13 pages, 3 figures, 5 tables

ODIN: Disentangled Reward Mitigates Hacking in RLHF

Lichang Chen, Chen Zhu, Davit Soselia, Jiuhai Chen, Tianyi Zhou, Tom Goldstein, Heng Huang, Mohammad Shoeybi, Bryan Catanzaro

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF

Banghua Zhu, Michael I. Jordan, Jiantao Jiao

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Eliciting Human Preferences with Language Models

Belinda Z. Li, Alex Tamkin, Noah Goodman, Jacob Andreas

专题命中 后训练与偏好优化 ：language model(title,abstract)；prompting(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 26 pages, 15 figures

Discovering Language Model Behaviors with Model-Written Evaluations

Ethan Perez, Sam Ringer, Kamilė Lukošiūtė, Karina Nguyen, Edwin Chen, Scott Heiner, Craig Pettit, Catherine Olsson, Sandipan Kundu, Saurav Kadavath, Andy Jones, Anna Chen, Ben Mann, Brian Israel, Bryan Seethor, Cameron McKinnon, Christopher Olah, Da Yan, Daniela Amodei, Dario Amodei, Dawn Drain, Dustin Li, Eli Tran-Johnson, Guro Khundadze, Jackson Kernion, James Landis, Jamie Kerr, Jared Mueller, Jeeyoon Hyun, Joshua Landau, Kamal Ndousse, Landon Goldberg, Liane Lovitt, Martin Lucas, Michael Sellitto, Miranda Zhang, Neerav Kingsland, Nelson Elhage, Nicholas Joseph, Noemí Mercado, Nova DasSarma, Oliver Rausch, Robin Larson, Sam McCandlish, Scott Johnston, Shauna Kravec, Sheer El Showk, Tamera Lanham, Timothy Telleen-Lawton, Tom Brown, Tom Henighan, Tristan Hume, Yuntao Bai, Zac Hatfield-Dodds, Jack Clark, Samuel R. Bowman, Amanda Askell, Roger Grosse, Danny Hernandez, Deep Ganguli, Evan Hubinger, Nicholas Schiefer, Jared Kaplan

专题命中 后训练与偏好优化 ：language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments for associated data visualizations, see https://www.evals.anthropic.com/model-written/ for full datasets, see https://github.com/anthropics/evals

Training Language Models with Language Feedback

Jérémy Scheurer, Jon Ander Campos, Jun Shern Chan, Angelica Chen, Kyunghyun Cho, Ethan Perez

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments The First Workshop on Learning with Natural Language Supervision at ACL 2022

Polaris: Learning to Generate Table Descriptions from Retrieval Feedback

Polaris：基于检索反馈学习生成表格描述

Ting Cai, Tuan Minh Phan, AnHai Doan

机构 * University of Wisconsin-Madison（威斯康星大学麦迪逊分校） 纠错

专题命中 后训练与偏好优化 ：LLM(summary_cn,abstract)；preference optimization(abstract)；分类 cs.CL

AI总结 Polaris是基于检索反馈训练LLM生成表格描述的系统，通过BM25排序和DPO微调优化检索效果，性能优于AutoDDG，证明检索基准可用于训练LLM生成面向检索的元数据。

Comments 22 pages, 6 figures

Why Summaries Turn Neutral: Policy Attribution for Sentiment Drift in Reinforcement Learning from Human Feedback

为何摘要变得中立：人类反馈强化学习中情感漂移的策略归因

Mikhail Krasitskii, Alexander Gelbukh, Olga Kolesnikova, Grigori Sidorov

机构 * Instituto Politécnico Nacional (IPN)（墨西哥国立理工学院） ； Centro de Investigación en Computación (CIC)（计算机研究中心） 纠错

专题命中 后训练与偏好优化 ：RLHF(summary_cn,abstract)；LLM(abstract_cn)；分类 cs.CL

AI总结 本文针对RLHF引发摘要情感漂移的问题，提出Policy Attribution框架溯源漂移来源，验证了跨语言漂移特性，并提出感知情感的正则化技术以降低漂移，且相关代码将公开。

CEDAR-GRPO: Process-Aware Reinforcement Learning for General Abductive Reasoning in LLMs

CEDAR-GRPO：面向大语言模型通用溯因推理的过程感知强化学习

Moein Salimi, Danial Parnian, Shaygan Adim, Amirmohammad Ebrahiminasab, Nima Alighardashi, Parsa Gholami, Sahand Akramipour, Mahdi Jafari Siavoshani, Mohammad Hossein Rohban

机构 * Sharif University of Technology（谢里夫理工大学） ； University of Tehran（德黑兰大学） 纠错

专题命中 后训练与偏好优化 ：LLM(summary_cn,abstract)；post-training(abstract)；分类 cs.AI

AI总结 本文提出过程感知强化学习框架CEDAR-GRPO，通过后训练提升LLM的通用溯因推理能力，在11个未见任务上实现显著性能提升，验证了其迁移有效性。

Comments Code and data are available at https://github.com/cedar-grpo/cedar-grpo

BridgeAlign: Bridging Preference Alignment for Humanities and Social Sciences

BridgeAlign：面向人文社科的偏好对齐框架

Ru Peng, Haokai Xu, Xijun Gu, Tianyu Zhao, Zhiting Fan, Yawen Zeng, Yihong Zhuang, Jinyang Zhang, Kexin Yang, Jian Wu, Hao Chen, Junyang Lin, Dayiheng Liu, Junbo Zhao

机构 * Alibaba Group（阿里巴巴集团） ； Ant Group（蚂蚁集团） 纠错

专题命中 后训练与偏好优化 ：LLM(abstract,abstract_cn)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

AI总结 该研究针对人文社科领域的偏好对齐需求，提出BridgeAlign框架，经21万+合成偏好样本对齐后，使Qwen3-8B在17个基准测试中优于11个强基线，且人工偏好与知识能力无权衡。

JustLLMGRPO: Radiographic Control for Chest X-Ray Generation

JustLLMGRPO：面向胸部X射线生成的放射学控制

Pengxiang Cai, Xiaohan Li, Anglin Liu, Qingyuan Zeng, Zexun Li, Jintai Chen

专题命中 后训练与偏好优化 ：LLM(summary_cn,abstract)；prompting(abstract)；分类 cs.AI

AI总结 JustLLMGRPO仅对LLM提示词策略应用GRPO，在冻结Sana生成器的前提下，提升了胸部X射线生成的质量，同时维持了提示词对齐度，实现了最优性能。

Teaching MLLMs to Say No: Generalized Referring Expression Comprehension via Refusal Calibrated GRPO

训练多模态大语言模型（MLLMs）说“不”：基于拒绝校准GRPO的广义指代表达理解

Xuzheng Yang, Jun Ling, Tao Huang, Caiyan Qin, Peng Wang

专题命中 后训练与偏好优化 ：SFT(abstract,abstract_cn)；large language model(abstract)；language model(abstract)；post-training(abstract)

AI总结 针对MLLMs因训练缺负样本导致无法拒绝不存在对象的问题，提出RC-GRPO方法，在保持定位精度的同时增强拒绝能力，在三个GREC基准上表现优异。

Evaluating Risks in Weak-to-Strong Alignment: A Bias-Variance Perspective

在弱到强对齐中评估风险：从偏差-方差视角出发

Hamid Osooli, Kareema Batool, Rick Gentry, Tiasa Singha Roy, Ashwin Gupta, Anirudha Ramesh

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； Microsoft（微软） ； InstaDeep ； New York University（纽约大学） 纠错

专题命中 后训练与偏好优化 ：SFT(abstract,abstract_cn)；RLHF(abstract,abstract_cn)；post-training(abstract)；分类 cs.AI

AI总结 本文从偏差-方差视角分析弱到强对齐的风险，通过连续置信度分数研究经验组件，发现强模型方差是欺骗的主要预测因素，表明弱强依赖性的重要性。

When Do Surrogate Updates Improve Decisions? A Local Theory of Trajectory-Wise Transfer

替代更新何时能改进决策？轨迹式迁移的局部理论

Yuyang Shen

机构 * The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）） 纠错

专题命中 后训练与偏好优化 ：LLM(summary_cn,abstract)；post-training(abstract)；分类 cs.LG

AI总结 该研究提出轨迹式迁移的局部理论，探究替代更新改进决策的条件，推导相关梯度、校准等理论结果，通过网格世界和LLM后训练实验验证结论。

Comments 22 pages in total, including references and appendices; 3 composite figures (9 panels) and 4 tables. Code is available at https://github.com/Ethan-Shen-Individual-Lab/surrogate-to-decision-transfer

Reinforcement Learning from Human Feedback

基于人类反馈的强化学习

Nathan Lambert

机构 * Nathan Lambert（纳瑟恩·拉姆伯特） 纠错

专题命中 后训练与偏好优化 ：RLHF(abstract,abstract_cn)；language model(abstract)；instruction tuning(abstract)；post-training(abstract)

AI总结 本书系统介绍强化学习从人类反馈的核心方法，涵盖指令微调、奖励模型训练及拒绝采样等关键技术，探讨合成数据与评估中的前沿问题。

Comments 239 pages. Web-native version at https://rlhfbook.com/ Continually improving, latest version at website

Agentic-DPO: From Imitation to Agentic Policy Optimization on Expert Trajectories

Agentic-DPO：从专家轨迹上的模仿到智能体策略优化

Yixiong Chen, Alan Yuille

机构 * Johns Hopkins University（约翰·霍普金斯大学） 纠错

专题命中 后训练与偏好优化 ：SFT(abstract,abstract_cn)；LLM(abstract)；large language model(abstract)；language model(abstract)

AI总结 研究针对大语言模型智能体基于专家轨迹训练只学动作序列、难应对错误的问题，提出Agentic-DPO方法，通过转化专家轨迹为状态条件偏好监督，结合策略保持增强，实现低成本智能体策略优化，实验验证其有效性。