语言大模型 / LLM

Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization

Shitong Duan, Xiaoyuan Yi, Peng Zhang, Yan Liu, Zheng Liu, Tun Lu, Xing Xie, Ning Gu

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted by EMNLP 2024(Findings)

A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models

Yi-Lin Tuan, William Yang Wang

专题命中 后训练与偏好优化 ：language model(title,abstract)；RLHF(abstract)；preference optimization(abstract)；分类 cs.CL、cs.LG

An Extremely Data-efficient and Generative LLM-based Reinforcement Learning Agent for Recommenders

Shuang Feng, Grace Feng

专题命中 后训练与偏好优化 ：LLM(title)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Understanding Reference Policies in Direct Preference Optimization

Yixin Liu, Pengfei Liu, Arman Cohan

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments GitHub Repo: https://github.com/yale-nlp/refdpo

Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimization

Zhanhui Zhou, Jie Liu, Jing Shao, Xiangyu Yue, Chao Yang, Wanli Ouyang, Yu Qiao

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments Findings of ACL 2024

A Long Way to Go: Investigating Length Correlations in RLHF

Prasann Singhal, Tanya Goyal, Jiacheng Xu, Greg Durrett

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments 21 pages, 13 figures, Accepted to COLM 2024

Prototypical Reward Network for Data-Efficient RLHF

Jinghan Zhang, Xiting Wang, Yiqiao Jin, Changyu Chen, Xinhao Zhang, Kunpeng Liu

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2024

Toward Optimal LLM Alignments Using Two-Player Games

Rui Zheng, Hongyi Guo, Zhihan Liu, Xiaoying Zhang, Yuanshun Yao, Xiaojun Xu, Zhaoran Wang, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang, Hang Li, Yang Liu

专题命中 后训练与偏好优化 ：LLM(title)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments Our code is released at https://github.com/ruizheng20/gpo

Creativity Has Left the Chat: The Price of Debiasing Language Models

Behnam Mohammadi

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Optimizing Language Models for Human Preferences is a Causal Inference Problem

Victoria Lin, Eli Ben-Michael, Louis-Philippe Morency

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.LG

Comments UAI 2024

Some things are more CRINGE than others: Iterative Preference Optimization with the Pairwise Cringe Loss

Jing Xu, Andrew Lee, Sainbayar Sukhbaatar, Jason Weston

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Aligning language models with human preferences

Tomasz Korbak

专题命中 后训练与偏好优化 ：language model(title,abstract)；pretraining(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments PhD thesis

Enhancing Autonomous Vehicle Training with Language Model Integration and Critical Scenario Generation

Hanlin Tian, Kethan Reddy, Yuxiang Feng, Mohammed Quddus, Yiannis Demiris, Panagiotis Angeloudis

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.AI、cs.LG

Comments 7 pages, 5 figures

Provably Robust DPO: Aligning Language Models with Noisy Feedback

Sayak Ray Chowdhury, Anush Kini, Nagarajan Natarajan

专题命中 后训练与偏好优化 ：language model(title,abstract)；SFT(abstract)；preference optimization(abstract)；分类 cs.CL、cs.LG

Configurable Safety Tuning of Language Models with Synthetic Preference Data

Victor Gallego

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI

Preference as Reward, Maximum Preference Optimization with Importance Sampling

Zaifan Jiang, Xing Huang, Chao Wei

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

ORPO: Monolithic Preference Optimization without Reference Model

Jiwoo Hong, Noah Lee, James Thorne

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；SFT(abstract)；分类 cs.CL、cs.AI

Comments Preprint

Direct Language Model Alignment from Online AI Feedback

Shangmin Guo, Biao Zhang, Tianlin Liu, Tianqi Liu, Misha Khalman, Felipe Llinares, Alexandre Rame, Thomas Mesnard, Yao Zhao, Bilal Piot, Johan Ferret, Mathieu Blondel

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 18 pages, 9 figures, 4 tables

Don't Forget Your Reward Values: Language Model Alignment via Value-based Calibration

Xin Mao, Feng-Lin Li, Huimin Xu, Wei Zhang, Anh Tuan Luu

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 19 pages, Under review

CovRL: Fuzzing JavaScript Engines with Coverage-Guided Reinforcement Learning for LLM-based Mutation

Jueon Eom, Seyeon Jeong, Taekyoung Kwon

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

Comments 14 pages, 4 figures, 9 tables, 2 listings

CycleAlign: Iterative Distillation from Black-box LLM to White-box Models for Better Human Alignment

Jixiang Hong, Quan Tu, Changyu Chen, Xing Gao, Ji Zhang, Rui Yan

专题命中 后训练与偏好优化 ：LLM(title)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Stabilizing RLHF through Advantage Model and Selective Rehearsal

Baolin Peng, Linfeng Song, Ye Tian, Lifeng Jin, Haitao Mi, Dong Yu

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments 9 pages, working in progress

HuatuoGPT, towards Taming Language Model to Be a Doctor

Hongbo Zhang, Junying Chen, Feng Jiang, Fei Yu, Zhihong Chen, Jianquan Li, Guiming Chen, Xiangbo Wu, Zhiyi Zhang, Qingying Xiao, Xiang Wan, Benyou Wang, Haizhou Li

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI

Position: The Complexity of Perfect AI Alignment -- Formalizing the RLHF Trilemma

位置：完美AI对齐的复杂性——形式化RLHF三重困境

Subramanyam Sahoo, Aman Chadha, Vinija Jain, Divya Chaudhary

机构 * Berkeley AI Safety Initiative (BASIS), University of California, Berkeley（伯克利人工智能安全倡议（BASIS），加州大学伯克利分校） ； AWS Generative AI Innovation Center, Amazon Web Services（亚马逊网络服务生成式人工智能创新中心） ； Meta AI ； Stanford University（斯坦福大学） ； Northeastern University, Seattle, WA, USA（东北大学，西雅图，华盛顿州，美国） 纠错

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

AI总结 研究提出RLHF三重困境，指出在安全、公平和稳健之间存在根本性权衡，并通过复杂性分析证明实现全球代表性需要超多项式计算资源。

Comments Accepted at NeurIPS 2025 Workshop on Socially Responsible and Trustworthy Foundation Models (ResponsibleFM)

Aligning Language Models Using Follow-up Likelihood as Reward Signal

Chen Zhang, Dading Chong, Feng Jiang, Chengguang Tang, Anningzhe Gao, Guohua Tang, Haizhou Li

专题命中 后训练与偏好优化 ：language model(title,abstract)；LLM(abstract,comments)；preference optimization(abstract)；分类 cs.CL

Comments Accepted by AAAI-2025, 16 pages, reward model, LLM Alignment, code repository at (https://github.com/e0397123/FLR)

GPT-Red: Automated Red Teaming via Self-Play at Scale

GPT-Red：基于大规模自博弈的自动化红队测试

Eric Wallace, Christopher A. Choquette-Choo, Nikhil Kandpal, Sam Toyer, Dylan Hunn, Stephanie Lin, Yuxin Wen, Xiangyu Qi, Christopher Wolff, Zizhao Wang, Milad Nasr, Sicheng Zhu, Chuan Guo, Juan Felipe Cerón Uribe, Kaiwen Wang, Aiden Low, Kai Xiao, Kai Chen

专题命中 后训练与偏好优化 ：LLM(summary_cn,abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 本研究推出GPT-Red，一种基于大规模自博弈的自动化红队测试智能体，可发现新型提示注入攻击、攻破过往模型且泛化能力强，用于提升LLM鲁棒性并形成自我改进飞轮。

Comments 28 pages.13 main pages and 13 main figures

Gradient Regularization Mitigates Reward Hacking in Reinforcement Learning from Human Feedback and Verifiable Rewards

梯度正则化减轻基于人类反馈和可验证奖励的强化学习中的奖励作弊

Johannes Ackermann, Michael Noukhovitch, Takashi Ishida, Masashi Sugiyama

机构 * The University of Tokyo（东京大学） ； RIKEN AIP（理化学研究所AIP） ； Mila（蒙特利尔大学Mila） 纠错

专题命中 后训练与偏好优化 ：RLHF(abstract,abstract_cn)；LLM(abstract)；language model(abstract)；post-training(abstract)

AI总结 研究基于人类反馈或可验证奖励的强化学习中奖励作弊问题，提出用梯度正则化使训练偏向奖励更准确区域，理论推导并实证验证，还改进方法，结果显示其比KL惩罚表现更好。

Comments Accepted at ICML 2026, 25 pages, 15 figures

Against the Monolithic Wireless World Model: Why NextG Needs Composable and Agentic Intelligence

反对单一无线世界模型：为什么NextG需要可组合和代理智能

Aladin Djuhera, Farhan Ahmed, Vlad C. Andrei, Swanand Ravindra Kadhe, Alecio Binotto, Haris Gacanin, Holger Boche

专题命中 后训练与偏好优化 ：LLM(summary_cn)；large language model(abstract)；language model(abstract)；foundation model(abstract)

AI总结 本文指出无线领域缺乏等同于LLM的数据基础，主张采用可组合和代理智能架构以实现可部署的AI原生网络。

Journal ref ICML 2026 Workshop on AI and ML for NextG

SPA: A SQL-Plan-Aware Reinforcement Learning Framework for Query Rewriting with LLMs

SPA：一种面向LLM查询重写的SQL计划感知强化学习框架

Xinyi Huang, Zhengjie Miao

专题命中 后训练与偏好优化 ：LLM(title_cn,summary_cn)

AI总结 提出SPA框架，利用物理执行反馈训练LLM重写SQL查询，通过概率门控自适应奖励塑形和策略自改进，显著提升端到端运行时性能并减少有害重写。

SWE-IF: Aligning Code Evaluation with Human Preference

SWE-IF: 使代码评估与人类偏好对齐

Ming Zhong, Xiang Zhou, Ting-Yun Chang, Qingze Wang, Nan Xu, Xiance Si, Dan Garrette, Shyam Upadhyay, Jeremiah Liu, Jiawei Han, Benoit Schillings, Jiao Sun

机构 * Google DeepMind（谷歌DeepMind） 纠错

专题命中 后训练与偏好优化 ：LLM(summary_cn,abstract_cn)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 提出SWE-IF基准，通过可验证指令分类法VeriCode评估代码指令遵循能力，发现指令遵循是区分LLM代码质量的关键，与功能正确性结合更能匹配人类偏好。

Comments ICML 2026