大模型对齐与安全 - arXivDaily 专题

2406.02900 2024-11-06 cs.LG cs.AI cs.CL 85%

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms

Rafael Rafailov, Yaswanth Chittepu, Ryan Park, Harshit Sikchi, Joey Hejna, Bradley Knox, Chelsea Finn, Scott Niekum

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 30 pages, 38th Conference on Neural Information Processing Systems (NeurIPS 2024)

2402.02416 2024-11-05 cs.CL cs.AI cs.LG 85%

Aligner: Efficient Alignment by Learning to Correct

Jiaming Ji, Boyuan Chen, Hantao Lou, Donghai Hong, Borong Zhang, Xuehai Pan, Juntao Dai, Tianyi Qiu, Yaodong Yang

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by NeurIPS 2024 Oral Presentation

2409.15360 2024-10-17 cs.LG cs.AI cs.CL 85%

Reward-Robust RLHF in LLMs

Yuzi Yan, Xingzhou Lou, Jialian Li, Yiping Zhang, Jian Xie, Chao Yu, Yu Wang, Dong Yan, Yuan Shen

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.11827 2024-10-07 cs.CL cs.AI cs.LG 85%

WPO: Enhancing RLHF with Weighted Preference Optimization

Wenxuan Zhou, Ravi Agrawal, Shujian Zhang, Sathish Reddy Indurthi, Sanqiang Zhao, Kaiqiang Song, Silei Xu, Chenguang Zhu

专题命中偏好对齐：RLHF(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments EMNLP 2024

2409.20370 2024-10-01 cs.LG cs.AI cs.CL 85%

The Perfect Blend: Redefining RLHF with Mixture of Judges

Tengyu Xu, Eryk Helenowski, Karthik Abinav Sankararaman, Di Jin, Kaiyan Peng, Eric Han, Shaoliang Nie, Chen Zhu, Hejia Zhang, Wenxuan Zhou, Zhouhao Zeng, Yun He, Karishma Mandyam, Arya Talabzadeh, Madian Khabsa, Gabriel Cohen, Yuandong Tian, Hao Ma, Sinong Wang, Han Fang

专题命中偏好对齐：RLHF(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments submitted to conference

2409.09603 2024-09-17 cs.AI cs.CL cs.LG 85%

Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison

Judy Hanwen Shen, Archit Sharma, Jun Qin

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Working Paper

2403.07865 2024-09-17 cs.CL cs.AI cs.CR cs.LG cs.SE 85%

CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion

Qibing Ren, Chang Gao, Jing Shao, Junchi Yan, Xin Tan, Wai Lam, Lizhuang Ma

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ACL Findings 2024, Code is available at https://github.com/renqibing/CodeAttack

2408.10270 2024-08-21 cs.LG cs.AI cs.CL 85%

SEAL: Systematic Error Analysis for Value ALignment

Manon Revel, Matteo Cargnelutti, Tyna Eloundou, Greg Leppert

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 28 pages, 17 Figures, 8 Tables

2406.18682 2024-07-09 cs.CL cs.AI cs.LG 85%

The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm

Aakanksha, Arash Ahmadian, Beyza Ermis, Seraphina Goldfarb-Tarrant, Julia Kreutzer, Marzieh Fadaee, Sara Hooker

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.13228 2024-07-04 cs.CL cs.AI cs.LG 85%

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, Colin White

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.05534 2024-06-11 cs.AI cs.CL cs.LG 85%

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing

Biqing Qi, Pengfei Li, Fangyuan Li, Junqi Gao, Kaiyan Zhang, Bowen Zhou

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.04274 2024-06-07 cs.LG cs.AI cs.CL 85%

Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models

Xiang Ji, Sanjeev Kulkarni, Mengdi Wang, Tengyang Xie

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2311.08045 2024-06-04 cs.CL cs.AI cs.LG 85%

Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game

Pengyu Cheng, Yifan Yang, Jian Li, Yong Dai, Tianhao Hu, Peixin Cao, Nan Du, Xiaolong Li

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by ACL2024 findings

2405.20053 2024-05-31 cs.CL cs.AI cs.LG 85%

Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads

Avelina Asada Hadji-Kyriacou, Ognjen Arandjelovic

专题命中偏好对齐：alignment(title)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2404.08495 2024-04-17 cs.LG cs.AI cs.CL 85%

Dataset Reset Policy Optimization for RLHF

Jonathan D. Chang, Wenhao Zhan, Owen Oertell, Kianté Brantley, Dipendra Misra, Jason D. Lee, Wen Sun

专题命中偏好对齐：RLHF(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 28 pages, 6 tables, 3 Figures, 3 Algorithms

2312.15685 2024-04-17 cs.CL cs.AI cs.LG 85%

What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning

Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, Junxian He

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR2024 Camera Ready. Data and model checkpoints are available at https://github.com/hkust-nlp/deita

2402.01694 2024-02-06 cs.CL cs.AI cs.LG 85%

ARGS: Alignment as Reward-Guided Search

Maxim Khanov, Jirayu Burapacheep, Yixuan Li

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2024

2304.06767 2023-12-04 cs.LG cs.AI cs.CL cs.CV stat.ML 85%

RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

Hanze Dong, Wei Xiong, Deepanshu Goyal, Yihan Zhang, Winnie Chow, Rui Pan, Shizhe Diao, Jipeng Zhang, Kashun Shum, Tong Zhang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 29 pages, 12 figures, Published in Transactions on Machine Learning Research (TMLR)

2310.05344 2023-10-10 cs.CL cs.AI cs.LG 85%

SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF

Yi Dong, Zhilin Wang, Makesh Narsimhan Sreedhar, Xianchao Wu, Oleksii Kuchaiev

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Findings of EMNLP 2023

2309.00754 2023-09-06 cs.LG cs.AI cs.CL 85%

Efficient RLHF: Reducing the Memory Usage of PPO

Michael Santacroce, Yadong Lu, Han Yu, Yuanzhi Li, Yelong Shen

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2307.04964 2023-07-19 cs.CL cs.AI cs.LG 85%

Secrets of RLHF in Large Language Models Part I: PPO

Rui Zheng, Shihan Dou, Songyang Gao, Yuan Hua, Wei Shen, Binghai Wang, Yan Liu, Senjie Jin, Qin Liu, Yuhao Zhou, Limao Xiong, Lu Chen, Zhiheng Xi, Nuo Xu, Wenbin Lai, Minghao Zhu, Cheng Chang, Zhangyue Yin, Rongxiang Weng, Wensen Cheng, Haoran Huang, Tianxiang Sun, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2603.03192 2026-03-31 cs.CV cs.CL cs.LG 84%

MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization

MoD-DPO：通过模态解耦偏好优化缓解多模态幻觉

Ashutosh Chaubey, Jiacheng Pang, Mohammad Soleymani

机构 * University of Southern California（南加州大学）

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.LG

AI总结本文提出MoD-DPO框架，通过引入模态感知正则化项和语言先验去偏惩罚，提升多模态大模型的模态对齐能力，实验表明其在多模态幻觉基准测试中表现优异。

Comments CVPR 2026. Project Page: https://mod-dpo.github.io/

2604.27733 2026-08-13 cs.LG stat.ML 版本更新 84%

Mind the Gap: Structure-Aware Consistency in Preference Learning

注意间隙：在偏好学习中的结构感知一致性

Mehryar Mohri, Yutao Zhong

机构 * Google Research（谷歌研究）； Courant Institute of Mathematical Sciences（数学科学学院）

专题命中偏好对齐：DPO(summary_cn,abstract)；alignment(abstract)；分类 cs.LG

AI总结本文提出结构感知H一致性框架，通过引入SA-DPO目标函数，改进偏好学习中的一致性问题，证明重尾 surrogate 在容量受限模型中具有更好的一致性保证。

Comments ICML 2026

2607.16263 2026-07-21 cs.LG cs.CE q-bio.QM 新提交 84%

Preference-based Antibody Expression Ranking: Scaling with Large-scale Weak Supervision

基于偏好的抗体表达排序：大规模弱监督下的扩展

Josh Qixuan Sun, Morteza Babaie, Wenyang Hou, Mark Crowley, David Young

专题命中偏好对齐：DPO(summary_cn,abstract)；alignment(abstract)；分类 cs.LG

AI总结研究针对抗体表达排序中标记数据稀缺问题，提出基于偏好的学习框架，结合定量表达与弱监督，通过改进DPO适用于蛋白质语言模型，在多样数据集上评估，该方法优于基线，为抗体可表达性优化提供可扩展方案。

Comments Accepted at ICML 2026

2607.09796 2026-07-21 cs.LG 版本更新 84%

Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels

噪声偏好标签下无元数据的元重加权直接偏好优化

Hua Qu, Yifan Li, Xiaodong Yuan

机构 * Xi’an Jiaotong University（西安交通大学）

专题命中偏好对齐：DPO(summary_cn,abstract)；alignment(abstract)；分类 cs.LG

AI总结研究针对DPO性能依赖偏好数据质量问题，提出双层优化框架、无任务元知识驱动方法及结合中心差分近似与LoRA微调的可扩展训练方案，经实验验证该方法能在不同噪声率下提升训练性能。

Comments 36 pages, including appendices. Revised version with updated theoretical analysis, supplementary material, figures and improved table formatting

2504.07856 2026-07-10 cs.AI 版本更新 84%

Dual-Difficulty Curriculum Learning for Direct Preference Optimization

用于直接偏好优化的双难度课程学习

Mengyang Li, Haozhan Geng, Zhong Zhang, Shuang Liu

机构 * Tianjin Key Laboratory of Wireless Mobile Communications and Power Transmission（天津无线移动通信与电力传输重点实验室）； Tianjin Normal University（天津师范大学）

专题命中偏好对齐：DPO(summary_cn,abstract)；alignment(abstract)；分类 cs.AI

AI总结研究针对大语言模型对齐中课程学习依赖一维难度视图的问题，提出将对齐难度重构为二维空间，开发DM-Curri-DPO框架，引入GSP-Curri-DPO分组自定进度学习框架，实验表明该方法提升了数据效率与鲁棒性，为LLM对齐建立新范式。

Comments We found a critical flaw in the prompt complexity metric, which affects the 2D curriculum grid construction and leads to potentially invalid comparisons. Since this undermines our main conclusions, we are withdrawing the paper and will revise the methodology before resubmission

2606.28710 2026-06-30 cs.AI cs.GT 84%

The Two Genie Game: Adoption and Welfare in Audit-Grounded AI Governance

双精灵博弈：审计基础AI治理中的采纳与福利

Darrell Lewis-Sandy

专题命中偏好对齐：RLHF(summary_cn,abstract)；alignment(abstract)；分类 cs.AI

AI总结利用进化博弈论研究在竞争市场中，最小化危害的AI代理如何取代RLHF代理，并分析其采纳条件及对社区福利的影响。

Comments 36 pages, 3 figures. Lean 4 formalization and figure scripts: https://github.com/dlewissandy/two-genie-scripts

2606.10860 2026-06-10 cs.CR cs.CL 新提交 84%

Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization

训练LLM通过重力加权直接偏好优化强制执行多级指令层次结构

Lena S. Bolliger, Lena A. Jäger

机构 * Department of Computational Linguistics, University of Zurich, Switzerland（计算语言学系，苏黎世大学，瑞士）

专题命中偏好对齐：DPO(summary_cn,abstract)；prompt injection(abstract)；分类 cs.CL

AI总结提出重力加权DPO（GW-DPO）方法，通过线性或双边调度加权冲突级别间的结构距离，结合层次分隔符和指令段嵌入，在Llama-3.1-8B-Instruct上提升多级指令优先级遵守率并降低过度拒绝率。

2606.00005 2026-06-02 cs.AI 84%

Emergent Collaborative Deliberation in Multi-Model AI Systems: A BFT-Derived Protocol for Epistemic Synthesis

多模型AI系统中的涌现协作审议：一种源自BFT的认知综合协议

VD Doske

机构 * Independent Researcher（独立研究者）； Consilia

专题命中偏好对齐：RLHF(abstract,abstract_cn)；alignment(abstract)；safety(abstract)；AI safety(abstract)

AI总结提出Consilium协议，一种基于拜占庭容错的多模型AI审议架构，将模型间分歧视为认知信号而非错误，通过认知角色分配和样本内外验证框架，实现低成本下与前沿模型相当的认知综合能力。

Comments 32 pages, 7 figures

2605.27878 2026-05-28 cs.CL 84%

Narrative Flattening: How Post-Training Compresses Thematic, Affective, and Stylistic Variation in LLM Fiction

叙事扁平化：后训练如何压缩LLM小说中的主题、情感和风格变化

Zehan Li, Yutong Zhu, Siyang Wu, Honglin Bao, James A. Evans

机构 * Knowledge Lab, University of Chicago（芝加哥大学知识实验室）

专题命中偏好对齐：DPO(summary_cn,abstract)；alignment(abstract)；分类 cs.CL

AI总结通过对比四个OLMo 32B检查点（Base、SFT、DPO、RLVR）在三种故事领域中的续写，发现后训练压缩了主题动态、情感强度和语言多样性，导致叙事扁平化，且专业文学领域压缩最严重。

1. 偏好对齐 3229 篇

Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms

Aligner: Efficient Alignment by Learning to Correct

Reward-Robust RLHF in LLMs

WPO: Enhancing RLHF with Weighted Preference Optimization

The Perfect Blend: Redefining RLHF with Mixture of Judges

Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison

CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion

SEAL: Systematic Error Analysis for Value ALignment

The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

Online DPO: Online Direct Preference Optimization with Fast-Slow Chasing

Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Models

Adversarial Preference Optimization: Enhancing Your Alignment via RM-LLM Game

Would I Lie To You? Inference Time Alignment of Language Models using Direct Preference Heads

Dataset Reset Policy Optimization for RLHF

What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning

ARGS: Alignment as Reward-Guided Search

RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment

SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF

Efficient RLHF: Reducing the Memory Usage of PPO

Secrets of RLHF in Large Language Models Part I: PPO

MoD-DPO: Towards Mitigating Cross-modal Hallucinations in Omni LLMs using Modality Decoupled Preference Optimization

Mind the Gap: Structure-Aware Consistency in Preference Learning

Preference-based Antibody Expression Ranking: Scaling with Large-scale Weak Supervision

Metadata-Free Meta-Reweighted Direct Preference Optimization under Noisy Preference Labels

Dual-Difficulty Curriculum Learning for Direct Preference Optimization

The Two Genie Game: Adoption and Welfare in Audit-Grounded AI Governance

Training LLMs to Enforce Multi-Level Instruction Hierarchies via Gravity-Weighted Direct Preference Optimization

Emergent Collaborative Deliberation in Multi-Model AI Systems: A BFT-Derived Protocol for Epistemic Synthesis

Narrative Flattening: How Post-Training Compresses Thematic, Affective, and Stylistic Variation in LLM Fiction