大模型对齐与安全 - arXivDaily 专题

2510.12044 2025-10-15 cs.CL cs.AI 84%

Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models

Yukun Zhang, Qi Dong

机构 * The Chinese University of Hong Kong（香港中文大学）； Fudan University（复旦大学）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2510.05410 2025-10-08 cs.CL cs.LG 84%

Aligning Language Models with Clinical Expertise: DPO for Heart Failure Nursing Documentation in Critical Care

Junyi Fan, Li Sun, Negin Ashrafi, Kamiar Alaei, Maryam Pishgar

机构 * University of Southern California（南加州大学）； California State University（加州州立大学）

专题命中偏好对齐：DPO(title,abstract)；safety(abstract)；分类 cs.CL、cs.LG

2509.24610 2025-10-01 cs.LG cs.CL 84%

OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment

Liang Lin, Zhihao Xu, Junhao Dong, Jian Zhao, Yuchen Yuan, Guibin Zhang, Miao Yu, Yiming Zhang, Zhengtao Yao, Huahui Yi, Dongrui Liu, Xinfeng Li, Kun Wang

机构 * Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究院（TeleAI），中国电信）； RUC（中国人民大学）； USTC（University of Science and Technology of China）； NTU（National University of Technology）； NUS（National University of Singapore）； USC（University of Southern California）； SCU（Sichuan University）； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.LG

2502.00666 2025-09-29 cs.LG cs.AI stat.ML 84%

Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration

Mingyu Chen, Yiding Chen, Wen Sun, Xuezhou Zhang

机构 * Department of Electrical & Computer Engineering（电气与计算机工程系）； Boston University（波士顿大学）； Department of Computer Science（计算机科学系）； Cornell University（康奈尔大学）； Faculty of Computing & Data Sciences（计算与数据科学学院）

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.AI、cs.LG

2508.21016 2025-08-29 cs.LG cs.AI 84%

Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance

Luozhijie Jin, Zijie Qiu, Jie Liu, Zijie Diao, Lifeng Qiao, Ning Ding, Alex Lamb, Xipeng Qiu

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2412.19792 2025-08-22 cs.LG cs.CL cs.IT math.IT 84%

InfAlign: Inference-aware language model alignment

Ananth Balashankar, Ziteng Sun, Jonathan Berant, Jacob Eisenstein, Michael Collins, Adrian Hutter, Jong Lee, Chirag Nagpal, Flavien Prost, Aradhana Sinha, Ananda Theertha Suresh, Ahmad Beirami

机构 * Google DeepMind（谷歌DeepMind）； Google Research（谷歌研究）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

2506.00845 2025-08-19 cs.LG cs.CL 84%

Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment

Yizhuo Zhang, Heng Wang, Shangbin Feng, Zhaoxuan Tan, Xinyun Liu, Yulia Tsvetkov

机构 * University of Washington（华盛顿大学）； Xi’an Jiaotong University（西安交通大学）； University of Notre Dame（圣母大学）； Google（谷歌）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments 8 pages, 1 figures, 2 tables. Experimental code and results are publicly available at https://anonymous.4open.science/r/Graph_RL-BF08/readme.md

2504.13134 2025-08-06 cs.CL cs.LG stat.ML 84%

Energy-Based Reward Models for Robust Language Model Alignment

Anamika Lochab, Ruqi Zhang

机构 * Department of Computer Science（计算机科学系）； Purdue University（普渡大学）

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL、cs.LG

Comments Accepted by COLM 2025

2406.16306 2025-08-05 cs.CL cs.LG stat.ML 84%

Cascade Reward Sampling for Efficient Decoding-Time Alignment

Bolian Li, Yifan Wang, Anamika Lochab, Ananth Grama, Ruqi Zhang

机构 * Department of Computer Science（计算机科学系）

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL、cs.LG

2502.05773 2025-07-28 cs.LG cs.AI stat.ML 84%

PIPA: Preference Alignment as Prior-Informed Statistical Estimation

Junbo Li, Zhangyang Wang, Qiang Liu

机构 * The University of Texas at Austin, US（德克萨斯大学奥斯汀分校）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2502.03699 2025-07-25 cs.CL cs.AI cs.IR 84%

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective

Bowen Jin, Jinsung Yoon, Zhen Qin, Ziqi Wang, Wei Xiong, Yu Meng, Jiawei Han, Sercan O. Arik

机构 * University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）； Google Cloud AI Research（谷歌云人工智能研究）； Google DeepMind（谷歌DeepMind）； University of Virginia（弗吉尼亚大学）

专题命中偏好对齐：alignment(title,abstract)；trustworthy(abstract)；分类 cs.CL、cs.AI

Comments 26 pages

2502.18699 2025-07-23 cs.CL cs.LG stat.ME 84%

MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment

Tianze Wang, Dongnan Gui, Yifan Hu, Shuhang Lin, Linjun Zhang

机构 * Department of Statistics, Rutgers University, New Brunswick, United States ； Department of Computer Science, Rutgers University, New Brunswick, United States ； College of Management of Technology, EPFL, Switzerland ； Department of Computer Science, ETH Zurich, Switzerland

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments ICML 2025

2507.07725 2025-07-11 cs.CL cs.AI 84%

Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization

Zhijin Dong

机构 * Peking University（北京大学）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2410.08316 2025-06-23 cs.LG cs.CL math.OC 84%

COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Framework

Yinuo Ren, Tesi Xiao, Michael Shavlovsky, Lexing Ying, Holakou Rahmanian

机构 * Institute for Computational and Mathematical Engineering (ICME), Stanford University（计算与数学工程研究所（ICME），斯坦福大学）； Amazon（亚马逊）； Department of Mathematics, Stanford University（数学系，斯坦福大学）

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.LG

Comments Published at UAI 2025

2411.06424 2025-06-10 cs.LG cs.CL 84%

How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis

Yushi Yang, Filip Sondej, Harry Mayne, Andrew Lee, Adam Mahdi

机构 * University of Oxford（牛津大学）； Jagiellonian University（雅盖隆大学）； Harvard University（哈佛大学）

专题命中偏好对齐：DPO(title,abstract)；safety(abstract)；分类 cs.CL、cs.LG

Journal ref NeurIPS 2024 Workshop on Socially Responsible Language Modelling Research (SoLaR)

2503.04647 2025-06-06 cs.CL cs.AI 84%

Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment

Wen Yang, Junhong Wu, Chen Wang, Chengqing Zong, Jiajun Zhang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Camera ready version for ACL 2025 Findings

2505.23729 2025-06-03 cs.CL cs.AI 84%

Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time

Mohamad Chehade, Soumya Suvra Ghosal, Souradip Chakraborty, Avinash Reddy, Dinesh Manocha, Hao Zhu, Amrit Singh Bedi

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICML 2025

2505.24369 2025-06-02 cs.LG cs.AI 84%

Adversarial Preference Learning for Robust LLM Alignment

Yuanfu Wang, Pengyu Wang, Chenyang Xi, Bo Tang, Junyi Zhu, Wenqiang Wei, Chen Chen, Chao Yang, Jingfeng Zhang, Chaochao Lu, Yijun Niu, Keming Mao, Zhiyu Li, Feiyu Xiong, Jie Hu, Mingchuan Yang

专题命中偏好对齐：alignment(title)；RLHF(abstract)；harmlessness(abstract)；分类 cs.AI、cs.LG

Comments Accepted at ACL2025 Findings

2501.09620 2025-05-30 cs.LG cs.AI 84%

Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment

Chaoqi Wang, Zhuokai Zhao, Yibo Jiang, Zhaorun Chen, Chen Zhu, Yuxin Chen, Jiayi Liu, Lizhu Zhang, Xiangjun Fan, Hao Ma, Sinong Wang

机构 * Meta ； University of Chicago（芝加哥大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2505.17558 2025-05-26 cs.CL cs.AI 84%

Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection

Shrey Pandit, Ashwin Vinod, Liu Leqi, Ying Ding

机构 * The University of Texas at Austin（德克萨斯大学奥斯汀分校）

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

Comments Code and dataset are available at https://teachingwithlies.github.io/

2503.10669 2025-05-20 cs.CL cs.AI 84%

UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality

Zelei Cheng, Xin-Qiang Cai, Yuting Tang, Pushi Zhang, Boming Yang, Masashi Sugiyama, Xinyu Xing

机构 * Northwestern University（西北大学）； RIKEN-AIP（日本理化学研究所-AIP）； The University of Tokyo（东京大学）； Microsoft Research Asia（微软亚洲研究院）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments Language Modeling, Machine Learning for NLP, Distributional Pareto-Optimal

2410.08067 2025-05-13 cs.LG cs.AI 84%

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs

Shenao Zhang, Zhihan Liu, Boyi Liu, Yufeng Zhang, Yingxiang Yang, Yongfei Liu, Liyu Chen, Tao Sun, Zhaoran Wang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments Published at ICML 2025

2411.00062 2025-04-11 cs.CL cs.AI physics.data-an stat.ML 84%

Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play

Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc V. Le, Qijun Tan, Yuan Liu

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments spotlight @ neurips language gamification workshop. updated the problem description and added new online RL experiments in this version

2411.02442 2025-04-01 cs.CL cs.AI cs.IR 84%

TODO: Enhancing LLM Alignment with Ternary Preferences

Yuxiang Guo, Lu Yin, Bo Jiang, Jiaqi Zhang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ICLR 2025

2410.01532 2025-04-01 cs.CL cs.AI cs.CV cs.HC 84%

Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models

Angela Lopez-Cardona, Carlos Segura, Alexandros Karatzoglou, Sergi Abadal, Ioannis Arapakis

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments This paper has been accepted to ICLR 2025

2410.12621 2025-03-26 cs.CL cs.LG 84%

Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoning

Ruimeng Ye, Yang Xiao, Bo Hui

专题命中偏好对齐：safety(title,abstract)；alignment(abstract)；分类 cs.CL、cs.LG

2503.13538 2025-03-19 cs.LG cs.AI 84%

From Demonstrations to Rewards: Alignment Without Explicit Human Preferences

Siliang Zeng, Yao Liu, Huzefa Rangwala, George Karypis, Mingyi Hong, Rasool Fakoor

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2502.18548 2025-03-14 cs.LG cs.AI 84%

What is the Alignment Objective of GRPO?

Milan Vojnovic, Se-Young Yun

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2503.09925 2025-03-14 cs.LG cs.CL 84%

PluralLLM: Pluralistic Alignment in LLMs via Federated Learning

Mahmoud Srewa, Tianyu Zhao, Salma Elmalaki

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

2503.03039 2025-03-06 cs.LG cs.AI 84%

LLM Misalignment via Adversarial RLHF Platforms

Erfan Entezami, Ali Naseh

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.AI、cs.LG

1. 偏好对齐 3229 篇

Hierarchical Alignment: Surgical Fine-Tuning via Functional Layer Specialization in Large Language Models

Aligning Language Models with Clinical Expertise: DPO for Heart Failure Nursing Documentation in Critical Care

OrthAlign: Orthogonal Subspace Decomposition for Non-Interfering Multi-Objective Alignment

Avoiding $\mathbf{exp(R_{max})}$ scaling in RLHF through Preference-based Exploration

Inference-Time Alignment Control for Diffusion Models with Reinforcement Learning Guidance

InfAlign: Inference-aware language model alignment

Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignment

Energy-Based Reward Models for Robust Language Model Alignment

Cascade Reward Sampling for Efficient Decoding-Time Alignment

PIPA: Preference Alignment as Prior-Informed Statistical Estimation

LLM Alignment as Retriever Optimization: An Information Retrieval Perspective

MPO: An Efficient Post-Processing Framework for Mixing Diverse Preference Alignment

Not All Preferences are What You Need for Post-Training: Selective Alignment Strategy for Preference Optimization

COS-DPO: Conditioned One-Shot Multi-Objective Fine-Tuning Framework

How Does DPO Reduce Toxicity? A Mechanistic Neuron-Level Analysis

Implicit Cross-Lingual Rewarding for Efficient Multilingual Preference Alignment

Bounded Rationality for LLMs: Satisficing Alignment at Inference-Time

Adversarial Preference Learning for Robust LLM Alignment

Beyond Reward Hacking: Causal Rewards for Large Language Model Alignment

Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection

UC-MOA: Utility-Conditioned Multi-Objective Alignment for Distributional Pareto-Optimality

Reward-Augmented Data Enhances Direct Preference Alignment of LLMs

Scalable Reinforcement Post-Training Beyond Static Human Prompts: Evolving Alignment via Asymmetric Self-Play

TODO: Enhancing LLM Alignment with Ternary Preferences

Seeing Eye to AI: Human Alignment via Gaze-Based Response Rewards for Large Language Models

Weak-to-Strong Generalization beyond Accuracy: a Pilot Study in Safety, Toxicity, and Legal Reasoning

From Demonstrations to Rewards: Alignment Without Explicit Human Preferences

What is the Alignment Objective of GRPO?

PluralLLM: Pluralistic Alignment in LLMs via Federated Learning

LLM Misalignment via Adversarial RLHF Platforms