大模型对齐与安全 - arXivDaily 专题

2603.10009 2026-03-12 cs.LG cs.AI cs.CL 85%

Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment

面向异质偏好对齐的个性化群体相对策略优化

Jialu Wang, Heinrich Peters, Asad A. Butt, Navid Hashemi, Alireza Hashemi, Pouya M. Ghari, Joseph Hoover, James Rae, Morteza Dehghani

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本文提出个性化GRPO框架，通过解耦优势估计与批次统计，提升模型对异质偏好信号的对齐能力。

2603.07017 2026-03-10 cs.CL cs.AI cs.LG 85%

Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models

安全能否从弱监督中涌现？小型语言模型的系统分析

Punyajoy Saha, Sudipta Halder, Debjyoti Mondal, Subhadarshi Panda

机构 * Samsung Research Institute（三星研究院）

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 Self-MOA通过弱监督实现小型语言模型的安全对齐，显著提升安全性的同时保持有用性，减少对人工标注数据的依赖。

Comments 19 pages, 10 tables, 7 figures, under Review

2503.18991 2026-03-03 cs.CL cs.AI cs.LG 85%

Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment

逆强化学习与动态奖励缩放用于大语言模型对齐

Ruoxi Cheng, Haoxuan Ma, Weixin Wang, Ranjie Duan, Jiexi Liu, Xiaoshuang Jia, Simeng Qin, Xiaochun Cao, Yang Liu, Xiaojun Jia

机构 * Beijing Electronic Science and Technology Institute（北京电子科技学院）； Alibaba Group（阿里巴巴集团）； Nanjing University（南京大学）； Duke University（杜克大学）； BraneMatrix AI ； Renmin University of China（中国人民大学）； Northeast University（东北大学）； Nanyang Technological University（南洋理工大学）； Zhejiang Lab（浙江实验室）； Sun Yat-sen University（中山大学）

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 DR-IRL通过动态奖励缩放和逆强化学习提升大语言模型的安全对齐性能，有效解决数据不平衡和静态奖励模型的局限性。

2510.03269 2026-02-18 cs.LG cs.AI cs.CL 85%

General Exploratory Bonus for Optimistic Exploration in RLHF

在RLHF中引入乐观探索的通用探索奖金

Wendi Li, Changdae Oh, Sharon Li

机构 * Department of Computer Sciences University of Wisconsin-Madison（计算机科学系威斯康星大学麦迪逊分校）

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本文提出GEB框架，通过理论分析和实验验证，解决了RLHF中乐观探索的偏差问题，提供了一种统一且有效的探索奖金方法。

Comments ICLR 2026

2502.06876 2026-02-03 cs.CL cs.AI cs.LG 85%

Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging

混合数据还是融合模型？通过模型融合平衡大型语言模型的有用性、诚实性和无害性

Jinluan Yang, Dingnan Jin, Anke Tang, Li Shen, Didi Zhu, Zhengyu Chen, Ziyu Zhao, Daixin Wang, Qing Cui, Zhiqiang Zhang, Jun Zhou, Fei Wu, Kun Kuang

专题命中偏好对齐：harmlessness(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本文提出RESM方法，通过改进的参数融合策略提升大型语言模型在有用性、诚实性和无害性方面的平衡对齐效果。

Comments arxiv version of NeurIPS2025

2601.22385 2026-02-02 cs.CL cs.AI cs.LG 85%

SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization

SP^2DPO: 一种基于大语言模型的语义逐对DPO泛化

Chaoyue He, Xin Zhou, Di Wang, Hong Xu, Wei Liu, Chunyan Miao

机构 * Alibaba--NTU Global e-Sustainability CorpLab (ANGEL), Singapore（阿里-NTU全球可持续性公司实验室（ANGEL），新加坡）； Alibaba Group, China（阿里集团，中国）

专题命中偏好对齐：DPO(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 SP^2DPO通过语义逐对DPO泛化提升AlpacaEval 2.0的长度控制胜率，避免每模型beta扫描。

Comments 39 pages, 15 figures, 16 tables, 60 equations

2601.08777 2026-01-14 cs.LG cs.AI cs.CL cs.GT 85%

Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling

渐近通用对齐：通过测试时间缩放实现的新对齐框架

Yang Cai, Weiqiang Zheng

机构 * Yale University（耶鲁大学）

专题命中偏好对齐：alignment(title,abstract)；trustworthy(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本文提出了一种基于测试时间缩放的通用对齐框架，通过多玩家对齐游戏实现最优鲁棒性，解决了现有方法在输出多样性上的不足。

2412.17417 2025-11-13 cs.CV 85%

Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment

Robert Wijaya, Ngoc-Bao Nguyen, Ngai-Man Cheung

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；safety(abstract)

2410.05102 2025-11-03 cs.CL cs.AI cs.LG 85%

SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks

Fenia Christopoulou, Ronald Cardenas, Gerasimos Lampouras, Haitham Bou-Ammar, Jun Wang

机构 * Huawei Noah’s Ark Lab（华为诺亚实验室）； University College London（伦敦大学学院）

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 27 pages, 9 figures, 5 tables. Accepted to EMNLP 2025

2504.09895 2025-10-14 cs.CL cs.AI cs.LG 85%

Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data

Shuai Zhao, Yunqiu Xu, Linchao Zhu, Yi Yang

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments The code is at https://github.com/mzhaoshuai/RefAlign

2405.11143 2025-10-10 cs.AI cs.CL cs.LG 85%

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

Jian Hu, Xibin Wu, Wei Shen, Jason Klein Liu, Zilin Zhu, Weixun Wang, Songlin Jiang, Haoran Wang, Hao Chen, Bin Chen, Weikai Fang, Xianyu, Yu Cao, Haotian Xu, Yiming Liu

机构 * Team Project Leader（团队项目负责人）

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments update template

2508.07768 2025-08-12 cs.LG cs.AI cs.CL 85%

Pareto Multi-Objective Alignment for Language Models

Qiang He, Setareh Maghsudi

机构 * Ruhr University Bochum（鲁尔大学波恩）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at ECML/PKDD 2025

2508.07750 2025-08-12 cs.LG cs.AI cs.CL 85%

Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment

Haowen Wang, Yun Yue, Zhiling Ye, Shuowen Zhang, Lei Fan, Jiaxin Liang, Jiadi Jiang, Cheng Wei, Jingyuan Deng, Xudong Han, Ji Li, Chunxiao Guo, Peng Wei, Jian Wang, Jinjie Gu

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 12 pages, 5 figures, 7 tables

2502.13417 2025-08-08 cs.CL cs.AI cs.LG 85%

RLTHF: Targeted Human Feedback for LLM Alignment

Yifei Xu, Tusher Chakraborty, Emre Kıcıman, Bibek Aryal, Eduardo Rodrigues, Srinagesh Sharma, Roberto Estevao, Maria Angels de Luis Balaguer, Jessica Wolk, Rafael Padilha, Leonardo Nunes, Shobana Balakrishnan, Songwu Lu, Ranveer Chandra

机构 * Microsoft（微软公司）； University of California, Los Angeles（加州大学洛杉矶分校）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Presented at ICML 2025

2412.04628 2025-06-23 cs.LG cs.AI cs.CL 85%

Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts

Taneesh Gupta, Rahul Madhavan, Xuchao Zhang, Nagarajan Natarajan, Chetan Bansal, Saravan Rajmohan

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2506.14903 2025-06-19 cs.CV 85%

DETONATE: A Benchmark for Text-to-Image Alignment and Kernelized Direct Preference Optimization

Renjith Prasad, Abhilekh Borah, Hasnat Md Abdullah, Chathurangi Shyalika, Gurpreet Singh, Ritvik Garimella, Rajarshi Roy, Harshul Surana, Nasrin Imanpour, Suranjana Trivedy, Amit Sheth, Amitava Das

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；safety(abstract)

Comments 59 pages, 10 figures

2410.02197 2025-06-12 cs.AI cs.CL cs.LG 85%

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment

Yifan Zhang, Ge Zhang, Yue Wu, Kangping Xu, Quanquan Gu

机构 * IIIS, Tsinghua University, Beijing, China（清华大学信息科学技术学院，北京，中国）； Shanghai Qi Zhi Institute, Shanghai, China（上海启智研究院，上海，中国）； Department of Computer Science, University of California, Los Angeles, California, USA（计算机科学系，加州大学洛杉矶分校，美国，加州）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to the 42nd International Conference on Machine Learning (ICML 2025)

2404.04656 2025-06-10 cs.LG cs.AI cs.CL 85%

Binary Classifier Optimization for Large Language Model Alignment

Seungjae Jung, Gunsoo Han, Daniel Wontae Nam, Kyoung-Woon On

机构 * Kakao Corp（Kakao公司）； LBOX

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ACL 2025 main

2402.10500 2025-06-10 cs.LG cs.AI cs.CL 85%

Active Preference Optimization for Sample Efficient RLHF

Nirjhar Das, Souradip Chakraborty, Aldo Pacchiano, Sayak Ray Chowdhury

机构 * Indian Institute of Science（印度科学研究院）； University of Maryland（马里兰大学）； Boston University（波士顿大学）； Indian Institute of Technology Kanpur（印度理工学院坎普尔分校）

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at ECML-PKDD 2025. Camera ready version

2404.16792 2025-06-02 cs.LG cs.AI cs.CL 85%

Model Extrapolation Expedites Alignment

Chujie Zheng, Ziqi Wang, Heng Ji, Minlie Huang, Nanyun Peng

机构 * The CoAI Group, DCST, BNRist, Tsinghua University（清华大学）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ACL 2025

2402.10184 2025-05-29 cs.LG cs.AI cs.CL cs.DM 85%

Reward Generalization in RLHF: A Topological Perspective

Tianyi Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Yang Han, Josef Dai, Xuehai Pan, Yaodong Yang

机构 * Center for AI Safety and Governance, Institute for AI, Peking University（人工智能安全与治理中心，人工智能研究院，北京大学）； Tsinghua University（清华大学）； Baichuan Inc（北川公司）

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 46 pages, ACL 2025 (Findings)

2502.09650 2025-05-15 cs.CL cs.AI cs.LG 85%

Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples

Chengqian Gao, Haonan Li, Liu Liu, Zeke Xie, Peilin Zhao, Zhiqiang Xu

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at ICML 2025

2410.04612 2025-04-25 cs.LG cs.AI cs.CL 85%

Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF

Zhaolin Gao, Wenhao Zhan, Jonathan D. Chang, Gokul Swamy, Kianté Brantley, Jason D. Lee, Wen Sun

机构 * Cornell University（康奈尔大学）； Princeton University（普林斯顿大学）； Databricks Mosaic Research（Databricks 混合研究）； Carnegie Mellon University（卡内基梅隆大学）； Harvard University（哈佛大学）

专题命中偏好对齐：RLHF(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2504.06659 2025-04-10 cs.LG cs.AI cs.CL 85%

Bridging the Gap Between Preference Alignment and Machine Unlearning

Xiaohua Feng, Yuyuan Li, Huwei Ji, Jiaming Zhang, Li Zhang, Tianyu Du, Chaochao Chen

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 17 pages

2502.16852 2025-02-25 cs.LG cs.AI cs.CL 85%

Improving LLM General Preference Alignment via Optimistic Online Mirror Descent

Yuheng Zhang, Dian Yu, Tao Ge, Linfeng Song, Zhichen Zeng, Haitao Mi, Nan Jiang, Dong Yu

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.04357 2025-02-10 cs.CL cs.AI cs.LG 85%

Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs

Hao Sun, Yunyi Shen, Jean-Francois Ton, Mihaela van der Schaar

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.04354 2025-02-10 cs.CL cs.AI cs.LG 85%

Reviving The Classics: Active Reward Modeling in Large Language Model Alignment

Yunyi Shen, Hao Sun, Jean-François Ton

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.07327 2025-02-10 cs.AI cs.CL cs.LG 85%

3D-Properties: Identifying Challenges in DPO and Charting a Path Forward

Yuzi Yan, Yibo Miao, Jialian Li, Yipin Zhang, Jian Xie, Zhijie Deng, Dong Yan

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Journal ref ICLR 2025

2412.14093 2024-12-23 cs.AI cs.CL cs.LG 85%

Alignment faking in large language models

Ryan Greenblatt, Carson Denison, Benjamin Wright, Fabien Roger, Monte MacDiarmid, Sam Marks, Johannes Treutlein, Tim Belonax, Jack Chen, David Duvenaud, Akbir Khan, Julian Michael, Sören Mindermann, Ethan Perez, Linda Petrini, Jonathan Uesato, Jared Kaplan, Buck Shlegeris, Samuel R. Bowman, Evan Hubinger

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

2412.04903 2024-12-17 cs.CV cs.AI cs.CL cs.LG 85%

EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation

Yongxin Wang, Meng Cao, Haokun Lin, Mingfei Han, Liang Ma, Jin Jiang, Yuhao Cheng, Xiaodan Liang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 19 pages

1. 偏好对齐 3229 篇

Personalized Group Relative Policy Optimization for Heterogenous Preference Alignment

Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models

Inverse Reinforcement Learning with Dynamic Reward Scaling for LLM Alignment

General Exploratory Bonus for Optimistic Exploration in RLHF

Mix Data or Merge Models? Balancing the Helpfulness, Honesty, and Harmlessness of Large Language Model via Model Merging

SP^2DPO: An LLM-assisted Semantic Per-Pair DPO Generalization

Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling

Synth-Align: Improving Trustworthiness in Vision-Language Model with Synthetic Preference Data Alignment

SparsePO: Controlling Preference Alignment of LLMs via Sparse Token Masks

Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Data

OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Framework

Pareto Multi-Objective Alignment for Language Models

Learning to Align, Aligning to Learn: A Unified Approach for Self-Optimized Alignment

RLTHF: Targeted Human Feedback for LLM Alignment

Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts

DETONATE: A Benchmark for Text-to-Image Alignment and Kernelized Direct Preference Optimization

Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment

Binary Classifier Optimization for Large Language Model Alignment

Active Preference Optimization for Sample Efficient RLHF

Model Extrapolation Expedites Alignment

Reward Generalization in RLHF: A Topological Perspective

Principled Data Selection for Alignment: The Hidden Risks of Difficult Examples

Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF

Bridging the Gap Between Preference Alignment and Machine Unlearning

Improving LLM General Preference Alignment via Optimistic Online Mirror Descent

Reusing Embeddings: Reproducible Reward Model Research in Large Language Model Alignment without GPUs

Reviving The Classics: Active Reward Modeling in Large Language Model Alignment

3D-Properties: Identifying Challenges in DPO and Charting a Path Forward

Alignment faking in large language models

EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation