大模型对齐与安全 - arXivDaily 专题

2506.02726 2025-06-04 cs.CL cs.AI cs.LG 87%

RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models

Qihang Yan, Xinyu Zhang, Luming Guo, Qi Zhang, Feifan Liu

机构 * ShanghaiTech University（上海科技大学）； Henan University（河南大学）； Liaoning University of Traditional Chinese Medicine（辽宁中医药大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.12678 2025-05-27 cs.LG cs.AI cs.CL 87%

Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees

Yongtao Wu, Luca Viano, Yihang Chen, Zhenyu Zhu, Kimon Antonakopoulos, Quanquan Gu, Volkan Cevher

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2410.01639 2025-05-13 cs.LG cs.AI cs.CY 87%

Moral Alignment for LLM Agents

Elizaveta Tennant, Stephen Hailes, Mirco Musolesi

机构 * University College London（伦敦大学学院）； University of Bologna（博洛尼亚大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.CY、cs.LG

Comments Published at the 13th International Conference on Learning Representations (ICLR'25), Singapore, Apr 2025. https://openreview.net/forum?id=MeGDmZjUXy

2410.13828 2025-04-23 cs.LG cs.AI cs.CL 87%

A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement

Hui Yuan, Yifan Zeng, Yue Wu, Huazheng Wang, Mengdi Wang, Liu Leqi

机构 * The University of Texas at Austin（德克萨斯大学奥斯汀分校）； Princeton Language & Intelligence（普林斯顿语言与智能）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

2408.15313 2025-04-09 cs.AI cs.CL cs.LG 87%

Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models

Wenxuan Zhang, Philip H. S. Torr, Mohamed Elhoseiny, Adel Bibi

专题命中偏好对齐：safety(title,abstract)；RLHF(abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments The paper has been accepted in ICLR 2025 as spotlight presentation

2410.19933 2025-02-28 cs.LG cs.AI cs.CY 87%

Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization

Xiyue Peng, Hengquan Guo, Jiawei Zhang, Dongqing Zou, Ziyu Shao, Honghao Wei, Xin Liu

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；harmlessness(abstract)；分类 cs.AI、cs.CY、cs.LG

2407.13399 2025-02-19 cs.AI cs.CL cs.LG 87%

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization

Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, Dylan J. Foster

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2501.03271 2025-01-22 cs.LG cs.AI cs.CL 87%

DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization

Amitava Das, Suranjana Trivedy, Danush Khanna, Rajarshi Roy, Gurpreet Singh, Basab Ghosh, Yaswanth Narsupalli, Vinija Jain, Vasu Sharma, Aishwarya Naresh Reganti, Aman Chadha

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

2411.14797 2024-11-25 cs.LG cs.AI cs.CL cs.CV 87%

Continual SFT Matches Multimodal RLHF with Negative Supervision

Ke Zhu, Yu Wang, Yanpeng Sun, Qiang Chen, Jiangjiang Liu, Gang Zhang, Jingdong Wang

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2411.01493 2024-11-12 cs.LG cs.AI cs.CL 87%

Sample-Efficient Alignment for LLMs

Zichen Liu, Changyu Chen, Chao Du, Wee Sun Lee, Min Lin

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2410.11677 2024-10-21 cs.CL cs.AI cs.LG 87%

Understanding Likelihood Over-optimisation in Direct Alignment Algorithms

Zhengyan Shi, Sander Land, Acyr Locatelli, Matthieu Geist, Max Bartolo

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint Version

2410.06293 2024-10-10 cs.LG cs.AI cs.CL 87%

Accelerated Preference Optimization for Large Language Model Alignment

Jiafan He, Huizhuo Yuan, Quanquan Gu

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 44 pages, 10 tables

2405.00675 2024-10-08 cs.LG cs.AI cs.CL stat.ML 87%

Self-Play Preference Optimization for Language Model Alignment

Yue Wu, Zhiqing Sun, Huizhuo Yuan, Kaixuan Ji, Yiming Yang, Quanquan Gu

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 27 pages, 4 figures, 5 tables

2405.01481 2024-09-04 cs.CL cs.AI cs.LG 87%

NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment

Gerald Shen, Zhilin Wang, Olivier Delalleau, Jiaqi Zeng, Yi Dong, Daniel Egert, Shengyang Sun, Jimmy Zhang, Sahil Jain, Ali Taghibakhshi, Markel Sanz Ausin, Ashwath Aithal, Oleksii Kuchaiev

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 16 pages, 4 figures, Accepted to COLM 2024

2406.15567 2024-06-25 cs.LG cs.AI cs.CL stat.ML 87%

SAIL: Self-Improving Efficient Online Alignment of Large Language Models

Mucong Ding, Souradip Chakraborty, Vibhu Agrawal, Zora Che, Alec Koppel, Mengdi Wang, Amrit Bedi, Furong Huang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 24 pages, 6 figures, 3 tables

2402.15018 2024-06-10 cs.CL cs.CY cs.LG 87%

Unintended Impacts of LLM Alignment on Global Representation

Michael J. Ryan, William Held, Diyi Yang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.CY、cs.LG

Comments Accepted to ACL 2024 main conference

2405.21046 2024-06-03 cs.LG cs.AI cs.CL stat.ML 87%

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

Tengyang Xie, Dylan J. Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, Alexander Rakhlin

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2404.00604 2024-04-02 cs.CL cs.AI cs.LG 87%

Extensive Self-Contrast Enables Feedback-Free Language Model Alignment

Xiao Liu, Xixuan Song, Yuxiao Dong, Jie Tang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2511.09105 2025-11-13 cs.LG cs.AI 87%

Cost-Minimized Label-Flipping Poisoning Attack to LLM Alignment

Shigeki Kusaka, Keita Saito, Mikoto Kudo, Takumi Tanabe, Akifumi Wachi, Youhei Akimoto

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments accepted for AAAI 2026 Special Track on AI Alignment

2506.06444 2025-07-10 cs.LG cs.AI cs.CR 87%

Saffron-1: Safety Inference Scaling

Ruizhong Qiu, Gaotang Li, Tianxin Wei, Jingrui He, Hanghang Tong

机构 * University of Illinois Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校）

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；jailbreak(abstract)；分类 cs.AI、cs.LG

Comments Previous title: "Saffron-1: Towards an Inference Scaling Paradigm for LLM Safety Assurance"

2402.08005 2024-02-14 cs.CL cs.LG 87%

Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs

Víctor Gallego

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；safety(abstract)；分类 cs.CL、cs.LG

Comments Pre-print. Submitted to the ICLR 2024 Workshop on Representational Alignment (Re-Align)

2606.08940 2026-06-09 cs.CL 新提交 86%

Multilingual Sentiment Aware Text Summarization A Reinforcement Learning Approach for Consistency Maintenance

多语言情感感知文本摘要：一种用于一致性维护的强化学习方法

Mikhail Krasitskii, Alexander Gelbukh, Olga Kolesnikova, Grigori Sidorov

机构 * Instituto Politécnico Nacional (IPN), Centro de Investigación en Computación (CIC)（国立理工学院（IPN），计算研究中心（CIC））

专题命中偏好对齐：RLHF(summary_cn,abstract)；alignment(abstract)；safety(abstract)；分类 cs.CL

AI总结研究RLHF摘要中的情感漂移现象，提出基于策略归因框架的情感感知KL正则化方法，在保持摘要质量的同时缓解情感中性化。

2407.16216 2026-05-19 cs.CL 86%

Reinforcement Learning for LLM Post-Training: A Survey

为大型语言模型进行后训练的强化学习：综述

Zhichao Wang, Kiran Ramnath, Bin Bi, Shiva Kumar Pentyala, Sougata Chaudhuri, Shubham Mehrotra, Zixu, Zhu, Xiang-Bo Mao, Sitaram Asur, Na, Cheng

机构 * Salesforce ； AWS AI Labs ； Airbnb

专题命中偏好对齐：RLHF(summary_cn,abstract)；DPO(abstract,abstract_cn)；分类 cs.CL

AI总结本文综述了通过强化学习进行大型语言模型后训练的方法，分析了RLHF和RLVR等技术，并提出统一的策略梯度框架，旨在为研究人员提供技术参考。

2604.24395 2026-04-28 cs.AI 86%

Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs

以自身声音对齐：用于减轻大型视觉-语言模型幻觉的自我纠正偏好学习

Byeonggeuk Lim, JungMin Yun, Junehyoung Kwon, Kyeonghyun Kim, YoungBin Kim

机构 * Graduate School of Advanced Imaging Sciences, Multimedia and Film（高级影像科学研究生院，多媒体与电影系）； Department of Artificial Intelligence（人工智能系）

专题命中偏好对齐：DPO(summary_cn,abstract)；alignment(abstract,abstract_cn)；分类 cs.AI

AI总结本文提出AVES-DPO框架，通过内在知识生成分布数据，利用共识验证机制诊断幻觉并引导模型自我纠正，有效缓解LVLMs的幻觉问题，仅需5200样本即优于现有基线。

Comments Accepted to ACL 2026

2507.16951 2025-07-24 cs.CL 86%

Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs

Shuyuan Lin, Lei Duan, Philip Hughes, Yuxuan Sheng

机构 * Sichuan University of Science and Engineering（四川理工学院）； Zagazig University（扎加齐亚大学）

专题命中偏好对齐：RLHF(title,abstract)；trustworthy(title)；分类 cs.CL

2605.11632 2026-06-05 cs.CL cs.AI 86%

Macro: Enhancing Multilingual Counterfactual Explanations through Alignment-as-Preference Optimization

Macro: 通过偏好对齐优化提升多语言反事实解释

Yilong Wang, Qianli Wang, Bohao Chu, Yihong Liu, Jing Yang, Simon Ostermann

机构 * Technische Universität Berlin（柏林技术大学）； German Research Center for Artificial Intelligence (DFKI)（德国人工智能研究中心）； University of Duisburg-Essen（杜伊斯堡- Essen大学）； LMU Munich（慕尼黑大学）； Munich Center for Machine Learning (MCML)（慕尼黑机器学习中心）； Saarland Informatics Campus（萨尔兰州信息学校区）； BIFOLD – Berlin Institute for the Foundations of Learning and Data（柏林学习与数据基础研究院）； Centre for European Research in Trusted AI (CERTAIN)（可信人工智能欧洲研究中心）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract,abstract_cn)；分类 cs.CL、cs.AI

AI总结本文提出Macro框架，通过直接偏好优化改进多语言反事实解释，提升有效性的同时保持最小性，避免翻译基线的严重最小性问题，并在多个指标上优于监督微调方法。

Comments In submission

2605.23961 2026-05-26 q-bio.BM cs.AI cs.LG 86%

Multimodal Alignment and Preference Optimization for Zero-Shot Conditional RNA Generation

多模态对齐与偏好优化用于零样本条件RNA生成

Roman Klypa, Alberto Bietti, Sergei Grudinin

机构 * Univ. Grenoble Alpes, CNRS, Grenoble INP, LJK（格勒诺布尔阿尔卑斯大学、法国国家科学研究中心、格勒诺布尔INP、LJK实验室）； Center for Computational Mathematics, Flatiron Institute（计算数学中心、Flatiron研究所）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract,abstract_cn)；分类 cs.AI、cs.LG

AI总结提出Moirain框架，通过多模态监督微调和直接偏好优化实现条件RNA序列生成，在零样本条件下生成具有高结合亲和力的生物合理RNA序列。

2605.17342 2026-05-19 cs.CL cs.AI 86%

Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment

传递性与循环性：动态大语言模型对齐的显式偏好分解

Yucong Huang, Xiucheng Li, Kaiqi Zhao, Jing Li

机构 * Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学深圳校区）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract,abstract_cn)；分类 cs.CL、cs.AI

AI总结本文提出Hybrid Reward-Cyclic模型，通过博弈论分解显式分离传递性和循环性偏好，结合动态自我博弈优化方法提升大语言模型对齐效果，实验证明其在混合传递-循环设置中具有结构优势和更高的准确率。

Comments Accepted by ICML 2026

2605.06375 2026-05-12 cs.LG cs.AI math.ST stat.TH 86%

A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment

一对GRPO家族的统一：从隐式到显式的偏好约束用于稳定和通用的RL对齐

Hao Yu

机构 * Department of Automation, Tsinghua University（清华大学自动化系）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract,abstract_cn)；分类 cs.AI、cs.LG

AI总结本文提出了一种统一的偏好基于RL优化框架，通过Soft-Pair-GRPO和Hard-Pair-GRPO两种变体，解决了主流配对偏好学习中政策更新不稳定、梯度方向模糊等问题，提升了对齐质量与训练稳定性。

2604.02652 2026-04-06 cs.LG cs.AI 86%

Generalization Limits of Reinforcement Learning Alignment

强化学习对齐的泛化极限

Haruhi Shida, Koo Imai, Keigo Kansa

机构 * Aladdin Security Inc（阿拉丁安全公司）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；safety(abstract)；分类 cs.AI、cs.LG

AI总结本文研究了强化学习对齐技术的泛化能力限制，提出复合越狱方法，通过结合多种攻击技术提高攻击成功率，验证了安全训练无法广泛泛化 hypothesis。

Comments 7 pages, 2 figures, 2 tables, accepted at JSAI 2026

1. 偏好对齐 3229 篇

RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models

Multi-Step Alignment as Markov Games: An Optimistic Online Gradient Descent Approach with Convergence Guarantees

Moral Alignment for LLM Agents

A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement

Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models

Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization

DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization

Continual SFT Matches Multimodal RLHF with Negative Supervision

Sample-Efficient Alignment for LLMs

Understanding Likelihood Over-optimisation in Direct Alignment Algorithms

Accelerated Preference Optimization for Large Language Model Alignment

Self-Play Preference Optimization for Language Model Alignment

NeMo-Aligner: Scalable Toolkit for Efficient Model Alignment

SAIL: Self-Improving Efficient Online Alignment of Large Language Models

Unintended Impacts of LLM Alignment on Global Representation

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

Extensive Self-Contrast Enables Feedback-Free Language Model Alignment

Cost-Minimized Label-Flipping Poisoning Attack to LLM Alignment

Saffron-1: Safety Inference Scaling

Refined Direct Preference Optimization with Synthetic Data for Behavioral Alignment of LLMs

Multilingual Sentiment Aware Text Summarization A Reinforcement Learning Approach for Consistency Maintenance

Reinforcement Learning for LLM Post-Training: A Survey

Aligning with Your Own Voice: Self-Corrected Preference Learning for Hallucination Mitigation in LVLMs

Harnessing RLHF for Robust Unanswerability Recognition and Trustworthy Response Generation in LLMs

Macro: Enhancing Multilingual Counterfactual Explanations through Alignment-as-Preference Optimization

Multimodal Alignment and Preference Optimization for Zero-Shot Conditional RNA Generation

Transitivity Meets Cyclicity: Explicit Preference Decomposition for Dynamic Large Language Model Alignment

A Unified Pair-GRPO Family: From Implicit to Explicit Preference Constraints for Stable and General RL Alignment

Generalization Limits of Reinforcement Learning Alignment