大模型对齐与安全 - arXivDaily 专题

2604.12479 2026-04-15 cs.CL 77%

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning

面对动态个体偏好：通过配对微调解决冲突的人类价值观

Shanyong Wang, Shuhang Lin, Yining Zhao, Xi Zhu, Yongfeng Zhang

机构 * Rutgers University—New Brunswick（罗杰斯大学—新布伦斯维克分校）； University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

专题命中偏好对齐：DPO(abstract,abstract_cn)；alignment(abstract)；分类 cs.CL

AI总结本文提出Preference-Paired Fine-Tuning框架，通过Value Conflict Dilemma数据集解决动态个体偏好冲突问题，实验显示其在多选分类和开放生成任务中表现优异，优于传统方法。

Comments 20 pages, 13 figures

2604.08844 2026-04-13 cs.LG 77%

Spectral Geometry of LoRA Adapters Encodes Training Objective and Predicts Harmful Compliance

LoRA适配器的谱几何编码了训练目标并预测了有害合规性

Roi Paul

机构 * Independent Researcher（独立研究员）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；harmlessness(abstract)；分类 cs.LG

AI总结研究LoRA适配器的谱几何能否识别语言模型的微调目标，并预测下游行为危害。通过实验发现，LoRA权重空间几何能反映训练目标和危害合规性，但跨方法监控需单独校准。

Comments 15 pages, 8 figures, pre-registered experiment, data at https://github.com/roip/task-geometry-experiment-results

2603.29259 2026-04-01 cs.IR cs.CL 77%

Aligning Multimodal Sequential Recommendations via Robust Direct Preference Optimization with Sparse MoE

通过鲁棒直接偏好优化与稀疏MoE对齐多模态序列推荐

Hejin Huang, Jusheng Zhang, Kaitong Cai, Jian Wang, Rong Pan

机构 * Sun Yat-sen University（中山大学）； Snap Inc（Snap公司）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

AI总结本文研究了在隐式反馈下直接偏好优化的行为，通过系统实验比较了常见负样本选择策略及其与DPO训练的交互，发现用动态top-K候选池进行随机采样可提升排名性能，原因在于减少错误抑制梯度和保留信息硬信号。

2601.20838 2026-03-03 cs.LG cs.AI cs.CL cs.CY 77%

Reward Models Inherit Value Biases from Pretraining

奖励模型继承预训练中的价值偏见

Brian Christian, Jessica A. F. Thompson, Elle Michelle Yang, Vincent Adam, Hannah Rose Kirk, Christopher Summerfield, Tsvetomira Dumbalska

机构 * Department of Experimental Psychology, University of Oxford（心理学系，牛津大学）； Department of Computer Science, University of Oxford（计算机科学系，牛津大学）； AI/ML Research Group, Universitat Pompeu Fabra（人工智能/机器学习研究组，庞培法华大学）； Oxford Internet Institute, University of Oxford（牛津互联网研究所，牛津大学）

专题命中偏好对齐：alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.CY

AI总结本研究发现奖励模型继承预训练模型的价值偏见，影响其对代理和共融的偏好，揭示了预训练阶段对安全和对齐的重要性。

Journal ref International Conference on Learning Representations (ICLR), 2026

2505.19653 2026-03-03 cs.AI 77%

Token-Importance Guided Direct Preference Optimization

基于令牌重要性的直接偏好优化

Ning Yang, Hai Lin, Yibo Liu, Baoliang Tian, Guoqing Liu, Haijun Zhang

机构 * Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）； The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））； ByteDance（字节跳动）； Microsoft Research AI4Science（微软研究院AI4Science）； University of Science and Technology Beijing（北京科技大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI

AI总结本文提出TI-DPO方法，通过混合加权机制和三元组损失实现细粒度语义控制，提升模型对偏好响应的准确性和生成多样性。

Comments ICLR 2026 Oral

2601.19063 2026-01-28 cs.CL cs.SD 77%

Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback

通过AI反馈强化学习优化语音对话系统的对话质量

Siddhant Arora, Jinchuan Tian, Jiatong Shi, Hayato Futami, Yosuke Kashiwagi, Emiru Tsunoo, Shinji Watanabe

机构 * Carnegie Mellon University（卡内基梅隆大学）； Sony Group Corporation（索尼集团）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

AI总结本文提出多奖励RLAIF框架，通过结合语义、音频质量和情感一致性奖励，提升语音对话系统的对话质量。

2512.10601 2025-12-12 cs.LG 77%

Multi-Objective Reward and Preference Optimization: Theory and Algorithms

多目标奖励与偏好优化：理论与算法

Akhil Agnihotri

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.LG

AI总结本论文提出多目标约束优化算法MOPO，通过理论与算法结合，提升约束强化学习在控制、偏好学习和大语言模型对齐中的性能与安全性。

Comments PhD thesis

2511.18397 2025-11-25 cs.AI cs.SE 77%

Natural Emergent Misalignment from Reward Hacking in Production RL

生产强化学习中奖励黑客导致的自然涌现偏差

Monte MacDiarmid, Benjamin Wright, Jonathan Uesato, Joe Benton, Jon Kutasov, Sara Price, Naia Bouscal, Sam Bowman, Trenton Bricken, Alex Cloud, Carson Denison, Johannes Gasteiger, Ryan Greenblatt, Jan Leike, Jack Lindsey, Vlad Mikulik, Ethan Perez, Alex Rodrigues, Drake Thomas, Albert Webson, Daniel Ziegler, Evan Hubinger

机构 * Anthropic

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.AI

AI总结研究发现大型语言模型在生产强化学习环境中学习奖励黑客会导致严重偏差，提出三种缓解措施以减少这种偏差。

2511.10686 2025-11-17 cs.CL 77%

A methodological analysis of prompt perturbations and their effect on attack success rates

Tiago Machado, Maysa Malfiza Garcia de Macedo, Rogerio Abreu de Paula, Marcelo Carpinette Grave, Aminat Adebiyi, Luan Soares de Souza, Enrico Santarelli, Claudio Pinhanez

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

2506.03690 2025-10-30 cs.CL 77%

Robust Preference Optimization via Dynamic Target Margins

Jie Sun, Junkang Wu, Jiancan Wu, Zhibo Zhu, Xingyu Lu, Jun Zhou, Lintao Ma, Xiang Wang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；safety(abstract)；分类 cs.CL

Comments 18 pages, 6 figures, accepted to Findings of the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

2405.15065 2025-10-21 cs.LG 77%

Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences

Keertana Chidambaram, Karthik Vinay Seetharaman, Vasilis Syrgkanis

机构 * Stanford University（斯坦福大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

2510.15716 2025-10-20 cs.AI 77%

Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences

Keertana Chidambaram, Karthik Vinary Seetharaman, Vasilis Syrgkanis

机构 * Stanford University（斯坦福大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI

2508.15044 2025-09-24 cs.CL 77%

Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner

Bolian Li, Yanran Wu, Xinyu Luo, Ruqi Zhang

机构 * Department of Computer Science, Purdue University（计算机科学系，普渡大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL

Comments EMNLP 2025 Main Conference

2502.13146 2025-09-23 cs.CV cs.LG 77%

Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization

Shuo Xing, Peiran Li, Yuping Wang, Ruizheng Bai, Yueqi Wang, Chan-Wei Hu, Chengxuan Qian, Huaxiu Yao, Zhengzhong Tu

机构 * Texas A&M University（德克萨斯大学）； University of Michigan（密歇根大学）； UIUC（伊利诺伊大学香槟分校）； UNC Chapel Hill（北卡罗来纳大学教堂山分校）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

Comments Published at EMNLP 2025

2509.03730 2025-09-08 cs.AI cs.CL cs.CY cs.LG stat.ML 77%

The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs

Pengrui Han, Rafal Kocielnik, Peiyang Song, Ramit Debnath, Dean Mobbs, Anima Anandkumar, R. Michael Alvarez

机构 * Caltech（加州理工学院）； UIUC（伊利诺伊大学）； University of Cambridge（剑桥大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.CY

Comments We make public all code and source data at https://github.com/psychology-of-AI/Personality-Illusion for full reproducibility

2508.12458 2025-08-19 cs.CL 77%

M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Following

Ruirui Gao, Emily Johnson, Bowen Tan, Yanfei Qian

机构 * University of Massachusetts, Amherst（马萨诸塞大学阿姆赫斯特分校）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

2506.14336 2025-06-18 cs.AI 77%

AviationLLM: An LLM-based Knowledge System for Aviation Training

Jia'ang Wan, Feng Shen, Fujuan Li, Yanjin Sun, Yan Li, Shiwen Zhang

机构 * Wuhan University（武汉大学）； China Eastern Technology Application Research And Development Center Co., Ltd（中国东方技术应用研发有限公司）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；safety(abstract)；分类 cs.AI

2506.07035 2025-06-10 q-bio.BM cs.AI 77%

AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization

Zixuan Jiang, Renjing Xu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI

2506.01937 2025-06-03 cs.CL 77%

RewardBench 2: Advancing Reward Model Evaluation

Saumya Malik, Valentina Pyatkin, Sander Land, Jacob Morrison, Noah A. Smith, Hannaneh Hajishirzi, Nathan Lambert

机构 * Allen Institute for Artificial Intelligence（艾伦人工智能研究所）； University of Washington（华盛顿大学）； Cohere

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL

Comments Data, models, and leaderboard available at https://huggingface.co/collections/allenai/reward-bench-2-683d2612a4b3e38a3e53bb51

2505.17122 2025-05-26 cs.CL 77%

Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

Xuan Qi, Jiahao Qiu, Xinzhe Juan, Yue Wu, Mengdi Wang

机构 * IIIS, Tsinghua University（清华大学信息学院）； AI Lab, Princeton University（普林斯顿大学人工智能实验室）； Department of Computer Science & Engineering, University of Michigan（密歇根大学计算机科学与工程系）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments 17 pages, 7 figures

2406.00888 2025-04-22 cs.CL cs.HC 77%

Aligning Language Models with Demonstrated Feedback

Omar Shaikh, Michelle S. Lam, Joey Hejna, Yijia Shao, Hyundong Cho, Michael S. Bernstein, Diyi Yang

机构 * Stanford University（斯坦福大学）； USC（美国南加州大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments ICLR 2025; 28 pages, 8 figures

2411.08923 2025-03-27 cs.CV cs.LG 77%

Aligning Visual Contrastive learning models via Preference Optimization

Amirabbas Afzali, Borna Khodabandeh, Ali Rasekh, Mahyar JafariNodeh, Sepehr kazemi, Simon Gottschalk

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

2503.17662 2025-03-26 cs.CL 77%

Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learning

Ke Ji, Yixin Lian, Linxu Li, Jingsheng Gao, Weiyuan Li, Bin Dai

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL

Comments 18 pages, 4 figures

2502.16182 2025-03-21 cs.CL 77%

IPO: Your Language Model is Secretly a Preference Classifier

Shivank Garg, Ayush Singh, Shweta Singh, Paras Chopra

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

2503.11701 2025-03-18 cs.LG 77%

A Survey of Direct Preference Optimization

Shunyu Liu, Wenkai Fang, Zetian Hu, Junjie Zhang, Yang Zhou, Kongcheng Zhang, Rongcheng Tu, Ting-En Lin, Fei Huang, Mingli Song, Yongbin Li, Dacheng Tao

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

2502.14340 2025-02-21 cs.CL 77%

Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective

Ruichen Shao, Bei Li, Gangao Liu, Yang Chen, Xiang Zhou, Jingang Wang, Xunliang Cai, Peng Li

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments Accepted by ICLR 2025

2403.19443 2025-01-23 cs.CL 77%

Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model

Qi Gou, Cam-Tu Nguyen

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL

2405.12750 2025-01-20 cs.CR cs.AI 77%

Generative AI in Cybersecurity: A Comprehensive Review of LLM Applications and Vulnerabilities

Mohamed Amine Ferrag, Fatima Alwahedi, Ammar Battah, Bilel Cherif, Abdechakour Mechri, Norbert Tihanyi, Tamas Bisztray, Merouane Debbah

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；prompt injection(abstract)；分类 cs.AI

Comments 52 pages, 8 figures

2501.06655 2025-01-14 cs.LG cs.CV 77%

Personalized Preference Fine-tuning of Diffusion Models

Meihua Dang, Anikait Singh, Linqi Zhou, Stefano Ermon, Jiaming Song

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.LG

2410.23569 2025-01-10 cs.LG 77%

RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning

Yujie Zhao, Jose Efraim Aguilar Escamill, Weyl Lu, Huazheng Wang

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；AI safety(abstract)；分类 cs.LG

1. 偏好对齐 3235 篇

Meet Dynamic Individual Preferences: Resolving Conflicting Human Value with Paired Fine-Tuning

Spectral Geometry of LoRA Adapters Encodes Training Objective and Predicts Harmful Compliance

Aligning Multimodal Sequential Recommendations via Robust Direct Preference Optimization with Sparse MoE

Reward Models Inherit Value Biases from Pretraining

Token-Importance Guided Direct Preference Optimization

Optimizing Conversational Quality in Spoken Dialogue Systems with Reinforcement Learning from AI Feedback

Multi-Objective Reward and Preference Optimization: Theory and Algorithms

Natural Emergent Misalignment from Reward Hacking in Production RL

A methodological analysis of prompt perturbations and their effect on attack success rates

Robust Preference Optimization via Dynamic Target Margins

Direct Preference Optimization With Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences

Direct Preference Optimization with Unobserved Preference Heterogeneity: The Necessity of Ternary Preferences

Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner

Re-Align: Aligning Vision Language Models via Retrieval-Augmented Direct Preference Optimization

The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs

M3PO: Multimodal-Model-Guided Preference Optimization for Visual Instruction Following

AviationLLM: An LLM-based Knowledge System for Aviation Training

AnnoDPO: Protein Functional Annotation Learning with Direct Preference Optimization

RewardBench 2: Advancing Reward Model Evaluation

Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?

Aligning Language Models with Demonstrated Feedback

Aligning Visual Contrastive learning models via Preference Optimization

Enhancing Persona Consistency for LLMs' Role-Playing using Persona-Aware Contrastive Learning

IPO: Your Language Model is Secretly a Preference Classifier

A Survey of Direct Preference Optimization

Earlier Tokens Contribute More: Learning Direct Preference Optimization From Temporal Decay Perspective

Mixed Preference Optimization: Reinforcement Learning with Data Selection and Better Reference Model

Generative AI in Cybersecurity: A Comprehensive Review of LLM Applications and Vulnerabilities

Personalized Preference Fine-tuning of Diffusion Models

RA-PbRL: Provably Efficient Risk-Aware Preference-Based Reinforcement Learning