大模型对齐与安全 - arXivDaily 专题

2406.10957 2024-12-10 cs.CL 77%

Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence

Junru Lu, Jiazheng Li, Siyu An, Meng Zhao, Yulan He, Di Yin, Xing Sun

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments EMNLP 2024 Main, Final Version

2410.16586 2024-10-23 cs.AI cs.IR 77%

Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective

Pietro Bernardelle, Gianluca Demartini

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI

2410.15096 2024-10-22 cs.AI 77%

GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets

Oh Joon Kwon, Daiki E. Matsunaga, Kee-Eung Kim

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI

Journal ref EMNLP 2024

2410.14184 2024-10-21 cs.CL 77%

MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time

Mozhi Zhang, Pengyu Wang, Chenkun Tan, Mianqiu Huang, Dong Zhang, Yaqian Zhou, Xipeng Qiu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments 19 pages, 6 figures

2404.12358 2024-08-14 cs.LG 77%

From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

Rafael Rafailov, Joey Hejna, Ryan Park, Chelsea Finn

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

Comments COLM 2024

2305.14718 2024-04-23 cs.CL 77%

Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models

Ashutosh Baheti, Ximing Lu, Faeze Brahman, Ronan Le Bras, Maarten Sap, Mark Riedl

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments published at ICLR 2024

2404.04291 2024-04-09 cs.LG 77%

Investigating Regularization of Self-Play Language Models

Reda Alami, Abdalgader Abubaker, Mastane Achab, Mohamed El Amine Seddik, Salem Lahlou

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

2402.14740 2024-02-27 cs.LG 77%

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

Comments 27 pages, 7 figures, 2 tables

2309.06657 2024-01-25 cs.CL 77%

Statistical Rejection Sampling Improves Preference Optimization

Tianqi Liu, Yao Zhao, Rishabh Joshi, Misha Khalman, Mohammad Saleh, Peter J. Liu, Jialu Liu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments Accepted in ICLR 2024

2605.18840 2026-05-26 cs.LG cs.AI cs.CL 76%

The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next

前沿模型的成长之痛：当排行榜不再区分以及接下来衡量什么

Adil Amin

机构 * Zehen Labs（泽亨实验室）

专题命中偏好对齐：RLHF(abstract,abstract_cn)；分类 cs.CL、cs.AI、cs.LG；alignment(comments)

AI总结本文通过分解SWE-bench和GPQA Diamond分数为种群耦合趋势和每版本残差（h场），诊断前沿模型能力之间的协作与权衡，并提供三步诊断法、每实验室测量优先级表及七个可证伪预测。

Comments 13 pages, 5 figures, 4 tables. Companion paper: "Lying Is Just a Phase: The Hidden Alignment Transition in Language Model Scaling." ( https://doi.org/10.48550/arXiv.2605.18838 ). Code: https://github.com/adilamin89/cape-scaling . Dashboard: https://zehenlabs.com/cape/

2505.23761 2025-10-03 cs.LG cs.AI cs.CL 76%

Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization

Yunjae Won, Hyunji Lee, Hyeonbin Hwang, Minjoon Seo

机构 * KAIST AI（韩国科学技术院人工智能研究所）

专题命中偏好对齐：DPO(abstract,comments)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint, under review. 39 pages, 12 figures. Updates from v1: Added new theoretical results on DPO training dynamics and policy exploration, included experiments with Qwen3-4B, and refined the discussion of log-margin dynamics

2411.07595 2025-06-16 cs.LG cs.AI cs.CL 76%

Entropy Controllable Direct Preference Optimization

Motoki Omura, Yasuhiro Fujita, Toshiki Kataoka

机构 * The University of Tokyo（东京大学）

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG；alignment(comments)

Comments ICML 2025 Workshop on Models of Human Feedback for AI Alignment

2603.26680 2026-05-12 cs.CL cs.AI 76%

AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment

AlpsBench: 一个面向真实对话记忆与偏好对齐的LLM个性化基准

Jianfei Xiao, Xiang Yu, Chengbing Wang, Wuqiang Zheng, Xinyu Lin, Kaining Liu, Hongxun Ding, Yang Zhang, Wenjie Wang, Fuli Feng, Xiangnan He

机构 * University of Science and Technology of China（科学技术大学）； National University of Singapore（新加坡国立大学）

专题命中偏好对齐：alignment(title)；分类 cs.CL、cs.AI

AI总结 AlpsBench通过真实人类与LLM对话数据构建，包含2500个长期交互序列及验证的记忆，评估个性化信息提取、更新、检索与利用等核心任务，揭示LLM在记忆管理中的局限性。

2506.05967 2026-05-12 cs.AI cs.LG stat.ML 76%

Preference Learning for AI Alignment: a Causal Perspective

偏好学习用于AI对齐：一种因果视角

Katarzyna Kobalczyk, Mihaela van der Schaar

机构 * Department of Applied Mathematics and Theoretical Physics（应用数学与理论物理系）

专题命中偏好对齐：alignment(title)；分类 cs.AI、cs.LG

AI总结本文从因果视角出发，探讨了基于偏好数据的奖励建模在对齐大语言模型与人类价值观中的关键作用，提出因果工具箱以解决因果误识别、偏好异质性和用户特定因素的混淆问题，并提出未来研究的方向。

Journal ref Proceedings of the 42nd International Conference on Machine Learning, Vancouver, Canada. PMLR 267, 2025

2505.18600 2026-04-13 cs.CV cs.AI cs.LG 76%

Chain-of-Zoom: Extreme Super-Resolution via Scale Autoregression and Preference Alignment

链式放大：通过尺度自回归和偏好对齐实现极端超分辨率

Bryan Sangwoo Kim, Jeongsol Kim, Jong Chul Ye

机构 * KAIST AI（韩国科学技术院人工智能系）

专题命中偏好对齐：alignment(title)；分类 cs.AI、cs.LG

AI总结本文提出Chain-of-Zoom框架，通过自回归链和多尺度提示实现极端超分辨率，利用GRPO优化文本提示对齐人类偏好，实验显示标准4x扩散模型在CoZ下可实现256倍以上高质量放大。

Comments NeurIPS 2025 (Spotlight)

2510.14628 2026-04-08 cs.CL cs.AI 76%

RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesis

RLAIF-SPA: 结构化AI反馈用于语音合成中的语义-语调对齐

Qing Yang, Zhenghao Liu, Yangfan Du, Pengcheng Huang, Tong Xiao

机构 * School of Computer Science and Engineering, Northeastern University, China（东北大学计算机科学与工程学院）

专题命中偏好对齐：alignment(title)；分类 cs.CL、cs.AI

AI总结本文提出RLAIF-SPA框架，通过整合强化学习从AI反馈来优化语音合成中的情感表达和可懂度，实验显示其在多个数据集上均取得显著提升。

2512.08777 2026-03-30 cs.CL cs.AI 76%

Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages

流畅对齐与不流畅评判：低资源语言的后训练方法

David Samuel, Lilja Øvrelid, Erik Velldal, Andrey Kutuzov

机构 * Language Technology Group, University of Oslo（奥斯陆大学语言技术组）

专题命中偏好对齐：alignment(title)；分类 cs.CL、cs.AI

AI总结本文提出一种低资源语言的后训练方法，通过不流畅评判模型保持语言模型的流畅性，通过挪威语案例研究验证了基于策略的训练方法在无需额外数据时的有效性。

Journal ref The Fourteenth International Conference on Learning Representations (ICLR 2026)

2504.09802 2025-11-04 cs.CL cs.AI 76%

Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignment

Wenrui Cai, Chengyu Wang, Junbing Yan, Jun Huang, Xiangzhong Fang

机构 * Shanghai Jiao Tong University（上海交通大学）； Alibaba Cloud Computing（阿里云计算）

专题命中偏好对齐：alignment(title)；分类 cs.CL、cs.AI

Comments emnlp 2025 main conference

2403.02745 2025-10-30 cs.AI cs.CL 76%

CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models

Son The Nguyen, Niranjan Uma Naresh, Theja Tulabandhula

机构 * University of Illinois Chicago（伊利诺伊大学芝加哥分校）； Independent Researcher（独立研究者）

专题命中偏好对齐：alignment(title)；分类 cs.CL、cs.AI

2510.14400 2025-10-21 cs.CL cs.AI cs.IR 76%

MedTrust-RAG: Evidence Verification and Trust Alignment for Biomedical Question Answering

Yingpeng Ning, Yuanyuan Sun, Ling Luo, Yanhua Wang, Yuchen Pan, Hongfei Lin

机构 * College of Computer Science and Technology, Dalian University of Technology（大连理工大学计算机科学与技术学院）； Air Force Communications NCO Academy（空军通信NCO学院）

专题命中偏好对齐：alignment(title)；分类 cs.CL、cs.AI

Comments Accepted as a short paper at BlBM2025

2510.05283 2025-10-08 cs.AI cs.CL cs.CV 76%

Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment

Radha Gulhane, Sathish Reddy Indurthi

机构 * Radha Gulhane（独立研究者）； Sathish Reddy Indurthi（独立研究者）

专题命中偏好对齐：alignment(title)；分类 cs.CL、cs.AI

2509.25568 2025-10-01 cs.CL cs.AI 76%

Probing the Limits of Stylistic Alignment in Vision-Language Models

Asma Farajidizaji, Akash Gupta, Vatsal Raina

机构 * Imperial College London（伦敦帝国学院）； Apta AI

专题命中偏好对齐：alignment(title)；分类 cs.CL、cs.AI

Comments 5 pages, 1 figure, 3 tables

2405.11870 2025-07-15 cs.CL cs.AI 76%

Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process

Ermo Hua, Biqing Qi, Kaiyan Zhang, Kai Tian, Xingtai Lv, Ning Ding, Bowen Zhou

机构 * Department of Electronic Engineering, Tsinghua University（清华大学电子工程系）； Shanghai AI Laboratory（上海人工智能实验室）

专题命中偏好对齐：alignment(title)；分类 cs.CL、cs.AI

Comments Accepted to ACL 2025, Oral & Panel Discussion

2505.23848 2025-06-02 cs.CL cs.LG 76%

Derailing Non-Answers via Logit Suppression at Output Subspace Boundaries in RLHF-Aligned Language Models

Harvey Dam, Jonas Knochelmann, Vinu Joseph, Ganesh Gopalakrishnan

专题命中偏好对齐：RLHF(title)；分类 cs.CL、cs.LG

2503.10460 2025-05-29 cs.CL cs.LG 76%

Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond

Liang Wen, Yunke Cai, Fenrui Xiao, Xin He, Qi An, Zhenyu Duan, Yimin Du, Junchen Liu, Lifu Tang, Xiaowei Lv, Haosheng Zou, Yongchao Deng, Shousheng Jia, Xiangzheng Zhang

机构 * First Author Affiliation（第一作者机构）

专题命中偏好对齐：DPO(title)；分类 cs.CL、cs.LG

Comments v4: ACL'25 industry track camera ready; v3: minor modifications; v2: better writing & format for later submission; all release at https://github.com/Qihoo360/Light-R1

2410.11410 2024-10-16 cs.CL cs.AI 76%

PMMT: Preference Alignment in Multilingual Machine Translation via LLM Distillation

Shuqiao Sun, Yutong Yao, Peiwen Wu, Feijun Jiang, Kaifu Zhang

专题命中偏好对齐：alignment(title)；分类 cs.CL、cs.AI

2305.17760 2024-01-03 cs.CL cs.LG 76%

Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective

Khanh Nguyen

专题命中偏好对齐：RLHF(title)；分类 cs.CL、cs.LG

Comments Proceedings of the First Workshop on Theory of Mind in Communicating Agents at (TOM @ ICML 2023)

2310.04373 2023-10-11 cs.LG cs.AI 76%

Confronting Reward Model Overoptimization with Constrained RLHF

Ted Moskovitz, Aaditya K. Singh, DJ Strouse, Tuomas Sandholm, Ruslan Salakhutdinov, Anca D. Dragan, Stephen McAleer

专题命中偏好对齐：RLHF(title)；分类 cs.AI、cs.LG

2209.14375 2022-09-30 cs.LG cs.CL 76%

Improving alignment of dialogue agents via targeted human judgements

Amelia Glaese, Nat McAleese, Maja Trębacz, John Aslanides, Vlad Firoiu, Timo Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker, Lucy Campbell-Gillingham, Jonathan Uesato, Po-Sen Huang, Ramona Comanescu, Fan Yang, Abigail See, Sumanth Dathathri, Rory Greig, Charlie Chen, Doug Fritz, Jaume Sanchez Elias, Richard Green, Soňa Mokrá, Nicholas Fernando, Boxi Wu, Rachel Foley, Susannah Young, Iason Gabriel, William Isaac, John Mellor, Demis Hassabis, Koray Kavukcuoglu, Lisa Anne Hendricks, Geoffrey Irving

专题命中偏好对齐：alignment(title)；分类 cs.CL、cs.LG

2503.04910 2025-03-10 cs.CL stat.ME 76%

Maximizing Signal in Human-Model Preference Alignment

Kelsey Kraus, Margaret Kroll

专题命中偏好对齐：alignment(title,comments)；分类 cs.CL

Comments Presented at AAAI 2025, special track on AI Alignment

1. 偏好对齐 3235 篇

Eliminating Biased Length Reliance of Direct Preference Optimization via Down-Sampled KL Divergence

Optimizing LLMs with Direct Preferences: A Data Efficiency Perspective

GDPO: Learning to Directly Align Language Models with Diversity Using GFlowNets

MetaAlign: Align Large Language Models with Diverse Preferences during Inference Time

From $r$ to $Q^*$: Your Language Model is Secretly a Q-Function

Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models

Investigating Regularization of Self-Play Language Models

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

Statistical Rejection Sampling Improves Preference Optimization

The Growing Pains of Frontier Models: When Leaderboards Stop Separating and What to Measure Next

Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization

Entropy Controllable Direct Preference Optimization

AlpsBench: An LLM Personalization Benchmark for Real-Dialogue Memorization and Preference Alignment

Preference Learning for AI Alignment: a Causal Perspective

Chain-of-Zoom: Extreme Super-Resolution via Scale Autoregression and Preference Alignment

RLAIF-SPA: Structured AI Feedback for Semantic-Prosodic Alignment in Speech Synthesis

Fluent Alignment with Disfluent Judges: Post-training for Lower-resource Languages

Enhancing Reasoning Abilities of Small LLMs with Cognitive Alignment

CURATRON: Complete and Robust Preference Data for Rigorous Alignment of Large Language Models

MedTrust-RAG: Evidence Verification and Trust Alignment for Biomedical Question Answering

Beyond Monolithic Rewards: A Hybrid and Multi-Aspect Reward Optimization for MLLM Alignment

Probing the Limits of Stylistic Alignment in Vision-Language Models

Intuitive Fine-Tuning: Towards Simplifying Alignment into a Single Process

Derailing Non-Answers via Logit Suppression at Output Subspace Boundaries in RLHF-Aligned Language Models

Light-R1: Curriculum SFT, DPO and RL for Long COT from Scratch and Beyond

PMMT: Preference Alignment in Multilingual Machine Translation via LLM Distillation

Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective

Confronting Reward Model Overoptimization with Constrained RLHF

Improving alignment of dialogue agents via targeted human judgements

Maximizing Signal in Human-Model Preference Alignment