大模型对齐与安全 - arXivDaily 专题

2603.22165 2026-03-24 cs.CV 82%

ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints

ACPO: 通过不对称约束对抗视觉-语言对齐中的似然位移

Kaili Huang, Hongming Zhang, Rui Shen, Linjun Dai, Jiahao Wang, Hanming Deng, Lewei Lu

机构 * SenseTime Research（商汤科技研究院）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

AI总结 ACPO通过动态目标导向缩放解决视觉-语言对齐中的似然位移问题，通过不对称抑制拒绝项梯度流，防止视觉锚点坍塌，提升多模态任务性能。

2603.18736 2026-03-20 cs.LG cs.AI cs.CL stat.ML 82%

CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks

CausalRM：基于观察性用户反馈的因果理论奖励建模用于RLHF

Hao Wang, Licheng Pan, Zhichao Chen, Chunyuan Zheng, Zhixuan Chu, Xiaoxi Li, Yuan Lu, Xinggao Liu, Haoxuan Li, Zhouchen Lin

机构 * Peking University（北京大学）； Xiaohongshu Inc.（小红书公司）； Zhejiang University（浙江大学）

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本文提出CausalRM框架，通过因果理论处理观察性用户反馈中的噪声和偏见问题，提升RLHF任务性能，实验显示在WildGuardMix和HarmBench上分别提升49.2%和32.7%。

2603.18411 2026-03-20 cs.CL cs.AI cs.LG 82%

TARo: Token-level Adaptive Routing for LLM Test-time Alignment

TARo: 令牌级自适应路由用于大语言模型测试时间对齐

Arushi Rai, Qiang Zhang, Hanqing Zeng, Yunkai Zhang, Dipesh Tamboli, Xiangjun Fan, Zhuokai Zhao, Lizhu Zhang

机构 * Meta ； University of Pittsburgh（匹兹堡大学）； University of California, Berkeley（加州大学伯克利分校）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 TARo通过令牌级自适应路由在推理时引导冻结的LLM进行结构化推理，提升推理性能达22.4%，并在医疗和指令遵循任务中表现优异。

2603.18008 2026-03-20 cs.CL cs.AI cs.CY 82%

TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots

TherapyGym: 评估和对齐疗法聊天机器人中的临床忠实性与安全性

Fangrui Huang, Souhad Chbeir, Arpandeep Khatua, Sheng Wang, Sijun Tan, Kenan Ye, Lily Bailey, Merryn Daniel, Ryan Louie, Sanmi Koyejo, Ehsan Adeli

机构 * Department of Computer Science, Stanford University, Stanford, CA, USA（计算机科学系，斯坦福大学，斯坦福，加州，美国）； Department of Psychiatry and Behavioral Sciences, Stanford University, Stanford, CA, USA（精神病学与行为科学系，斯坦福大学，斯坦福，加州，美国）； Department of Biomedical Data Science, Stanford University, Stanford, CA, USA（生物医学数据科学系，斯坦福大学，斯坦福，加州，美国）； University of California, Berkeley, Berkeley, CA, USA（加州大学伯克利分校，伯克利，加州，美国）； The University of Hong Kong, Hong Kong（香港大学，香港）

专题命中偏好对齐：safety(title,abstract)；分类 cs.CL、cs.AI、cs.CY

AI总结 TherapyGym通过评估和提升疗法聊天机器人中的忠实性与安全性，结合CTRS评分和多标签标注方案，利用RL训练框架改进模型性能，提升临床应用的安全性和有效性。

2602.19442 2026-03-12 cs.CV 82%

UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment

UrbanAlign: 域内任务中VLM与人类偏好对齐的后处理语义校准

Yecheng Zhang, Rong Zhao, Zhizhou Sha, Yong Li, Lei Wang, Ce Hou, Wen Ji, Hao Huang, Yunshan Wan, Jian Yu, Junhao Xia, Yuru Zhang, Chunlei Shi

机构 * Tsinghua University（清华大学）； University College London（伦敦大学学院）； Hong Kong University of Science and Technology（香港科学与技术大学）； Peking University（北京大学）； Southwest Jiaotong University（西南交通大学）； Zhejiang University（浙江大学）； University of Texas at Austin（德克萨斯大学奥斯汀分校）； Renmin University of China（中国人民大学）； Southeast University（东南大学）

专题命中偏好对齐：alignment(title)；RLHF(abstract)；safety(abstract)

AI总结 UrbanAlign通过后处理方法实现VLM与人类偏好的对齐，无需修改模型权重，提升领域任务的准确性和可解释性。

Comments 26 pages

2509.23371 2026-03-02 cs.CL cs.AI cs.LG 82%

Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization

通过元权重在线采样对齐：弥合数据生成与偏好优化之间的差距

Junming Yang, Ning Xu, Biao Liu, Shiqi Qiao, Xin Geng

机构 * School of Computer Science and Engineering, Southeast University, Nanjing, China（东南大学计算机科学与工程学院）； Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University), Ministry of Education, China（新一代人工智能技术及其交叉应用国家重点实验室）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 MetaAPO通过动态结合数据生成与模型训练，有效解决偏好优化中的分布不匹配问题，提升对齐效果并降低标注成本。

Comments Accepted by ICLR 2026

2602.16802 2026-02-20 cs.CL cs.AI cs.LG 82%

References Improve LLM Alignment in Non-Verifiable Domains

参考文献提高非可验证领域的LLM对齐

Kejian Shi, Yixin Liu, Peifeng Wang, Alexander R. Fabbri, Shafiq Joty, Arman Cohan

机构 * Yale University（耶鲁大学）； Meta ； Scale AI ； Salesforce Research（Salesforce 研究）； Nanyang Technological University（南洋理工大学）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本研究提出参考引导的LLM评估器，通过增强LLM对齐的评估器和自我改进，显著提升了非可验证领域中LLM的性能。

Comments ICLR 2026 Camera Ready

2502.14560 2026-02-17 cs.LG cs.AI cs.CL 82%

Less is More: Improving LLM Alignment via Preference Data Selection

少即是多：通过偏好数据选择改进大语言模型对齐

Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang, Xiangnan He

机构 * University of Science and Technology of China（中国科学技术大学）； Peking University（北京大学）； Massachusetts Institute of Technology（麻省理工学院）； Alibaba Cloud Computing（阿里云计算）； MoE Key Lab of BIPC, University of Science and Technology of China（中国科学技术大学MoE关键实验室）

专题命中偏好对齐：alignment(title)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结通过改进偏好数据选择策略，提升大语言模型对齐效果，实验显示在较少数据下实现显著性能提升。

2602.03665 2026-02-04 cs.CV cs.HC 82%

MM-SCALE: Grounded Multimodal Moral Reasoning via Scalar Judgment and Listwise Alignment

MM-SCALE: 基于标量判断和列表对齐的 grounded 多模态道德推理

Eunkyu Park, Wesley Hanwen Deng, Cheyon Jin, Matheus Kunzler Maldaner, Jordan Wheeler, Jason I. Hong, Hong Shen, Adam Perer, Ken Holstein, Motahhare Eslami, Gunhee Kim

机构 * Seoul National University（首尔国立大学）； Carnegie Mellon University（卡内基梅隆大学）； University of Florida（佛罗里达大学）； Epic Games

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)

AI总结 MM-SCALE通过5点标量评分和显式模态接地，提升多模态模型在道德推理任务中的表现。

2601.20433 2026-02-02 cs.CV 82%

MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models

MARE: 多模态对齐与强化学习用于通过视觉-语言模型的可解释深度伪造检测

Wenbo Xu, Wei Lu, Xiangyang Luo, Jiantao Zhou

机构 * School of Computer Science and Engineering, MoE Key Laboratory of Information Technology, Guangdong Province Key Laboratory of Information Security Technology, Sun Yat-sen University, Guangzhou 510006, China（计算机科学与工程学院，信息技术MOE实验室，广东省信息安全技术重点实验室，中山大学，广州510006，中国）； State Key Laboratory of Mathematical Engineering and Advanced Computing（数学工程与先进计算国家重点实验室）； Department of Computer and Information Science, University of Macau.（计算机与信息科学系，澳门大学）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)

AI总结 MARE通过多模态对齐和强化学习提升视觉-语言模型在深度伪造检测中的准确性和可靠性。

2601.12758 2026-01-21 cs.CL cs.AI cs.LG 82%

VISPA: Pluralistic Alignment via Automatic Value Selection and Activation

VISPA：通过自动价值选择和激活实现多元对齐

Shenyan Zheng, Jiayou Zhong, Anudeex Shetty, Heng Ji, Preslav Nakov, Usman Naseem

机构 * University of Waterloo（滑铁卢大学）； University of Melbourne（墨尔本大学）； University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）； MBZUAI（马克斯·普朗克智能系统研究所）； Macquarie University（麦考瑞大学）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 VISPA通过自动价值选择和激活实现多元对齐，适用于多种模型和场景，提供了一种可扩展的语言模型对齐方法。

Comments WIP

2601.06180 2026-01-13 cs.LG cs.AI cs.CL 82%

MixDPO: Modeling Preference Strength for Pluralistic Alignment

MixDPO：建模偏好强度以实现多元对齐

Saki Imai, Pedram Heydari, Anthony Sicilia, Asteria Kaeberlein, Katherine Atwell, Malihe Alikhani

机构 * Northeastern University（东北大学）； Johns Hopkins University（约翰霍普金斯大学）； West Virginia University（西弗吉尼亚大学）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 MixDPO通过建模偏好强度差异，提升多样的偏好对齐性能，同时保持子群体偏好，适用于高异质性场景。

2506.06970 2026-01-01 cs.CV 82%

Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment

通过偏好对齐引导跨模态表示

Pengfei Zhao, Rongbo Luan, Wei Zhang, Peng Wu, Sifeng He

机构 * Apple（苹果公司）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

AI总结 MAPLE通过利用多模态大语言模型的细粒度对齐先验，引导跨模态表示学习，提升细粒度检索性能。

Comments Accepted by NeurIPS 2025

2511.11255 2025-11-25 cs.IR 82%

Align$^3$GR: Unified Multi-Level Alignment for LLM-based Generative Recommendation

Align$^3$GR: 基于大语言模型的生成推荐系统统一多级对齐

Wencai Ye, Mingjie Sun, Shuhang Chen, Wenjin Wu, Peng Jiang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

AI总结 Align$^3$GR通过统一多级对齐方法，提升大语言模型在生成推荐系统中的性能，实验显示在Recall@10和NDCG@10指标上均取得显著提升。

Comments Accepted by AAAI 2026 (Oral)

2507.20067 2025-11-14 cs.AI cs.CL cs.LG 82%

PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training

Sarat Chandra Bobbili, Ujwal Dinesha, Dheeraj Narasimha, Srinivas Shakkottai

机构 * Texas A&M University（德克萨斯A&M大学）； Inria（法国国家信息与自动化研究所）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2511.03997 2025-11-07 cs.CV 82%

PhysCorr: Dual-Reward DPO for Physics-Constrained Text-to-Video Generation with Automated Preference Selection

Peiyao Wang, Weining Wang, Qi Li

专题命中偏好对齐：DPO(title)；alignment(abstract)；trustworthy(abstract)

2511.01083 2025-11-04 cs.RO 82%

Deployable Vision-driven UAV River Navigation via Human-in-the-loop Preference Alignment

Zihan Wang, Jianwen Li, Li-Fan Wu, Nina Mahmoudian

机构 * School of Mechanical Engineering, Purdue University（机械工程学院，普渡大学）

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)

Comments Submitted to ICRA 2026

2510.27254 2025-11-03 cs.CL cs.AI cs.LG 82%

Languages are Modalities: Cross-Lingual Alignment via Encoder Injection

Rajan Agarwal, Aarush Gupta

机构 * University of Waterloo（多伦多大学）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 14 pages, 3 Figures

2505.05465 2025-10-28 cs.CL cs.AI cs.LG 82%

ComPO: Preference Alignment via Comparison Oracles

Peter Chen, Xi Chen, Wotao Yin, Tianyi Lin

机构 * Columbia University（哥伦比亚大学）； Stern School of Business（斯特恩商学院）； New York University（纽约大学）； DAMO Academy, Alibaba Group US（阿里云达摩院）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to NeurIPS 2025

2505.18126 2025-09-30 cs.LG cs.AI cs.CL 82%

Reward Model Overoptimisation in Iterated RLHF

Lorenz Wolf, Robert Kirk, Mirco Musolesi

机构 * UCL Centre for Artificial Intelligence（伦敦大学学院人工智能中心）； Department of Computer Science（计算机科学系）； University College London（伦敦大学学院）； UK AI Security Institute（英国人工智能安全研究所）； Department of Computer Science and Engineering（计算机科学与工程系）； University of Bologna（博洛尼亚大学）

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 24 pages, 17 figures, 6 tables

2504.12083 2025-09-30 cs.CV 82%

Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization

Pritam Sarkar, Ali Etemad

机构 * Queen’s University（女王大学）； Vector Institute（向量研究所）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

Comments NeurIPS 2025

2509.20938 2025-09-26 cs.RO cs.CV 82%

Autoregressive End-to-End Planning with Time-Invariant Spatial Alignment and Multi-Objective Policy Refinement

Jianbo Zhao, Taiyu Ban, Xiangjie Li, Xingtai Gui, Hangning Zhou, Lei Liu, Hongwei Zhao, Bin Li

机构 * University of Science and Technology of China（中国科学技术大学）； Mach Drive（马车驱动）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

2508.05165 2025-08-08 cs.LG cs.AI cs.CL 82%

Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models

Mason Nakamura, Saaduddin Mahmud, Kyle H. Wray, Hamed Zamani, Shlomo Zilberstein

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2507.20133 2025-07-30 cs.CL cs.AI cs.LG 82%

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

Anas Mohamed, Azal Ahmad Khan, Xinran Wang, Ahmad Faraz Khan, Shuwen Ge, Saman Bahzad Khan, Ayaan Ahmad, Ali Anwar

机构 * University of Minnesota（明尼苏达大学）； Virginia Tech（弗吉尼亚理工大学）； Xi’an University of Technology（西安理工大学）； Lahore University of Management Sciences（拉合尔管理科学大学）； University of California, Santa Cruz（加州大学圣克鲁兹分校）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2507.00018 2025-07-08 cs.LG cs.AI cs.CL 82%

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

Bo Wang, Qinyuan Cheng, Runyu Peng, Rong Bao, Peiji Li, Qipeng Guo, Linyang Li, Zhiyuan Zeng, Yunhua Zhou, Xipeng Qiu

机构 * School of Computer Science, Fudan University（复旦大学计算机学院）； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2507.01492 2025-07-03 cs.CV 82%

AVC-DPO: Aligned Video Captioning via Direct Preference Optimization

Jiyang Tang, Hengyi Li, Yifan Du, Wayne Xin Zhao

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）； College of Artificial Intelligence, Nankai University（南开大学人工智能学院）； School of Computer Science and Technology, Beijing Institute of Technology（北京理工大学计算机科学与技术学院）

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)

2506.15717 2025-06-23 cs.LG cs.AI cs.CL 82%

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities

Zhengze Zhang, Shiqi Wang, Yiqun Shen, Simin Guo, Dahua Lin, Xiaoliang Wang, Nguyen Cam-Tu, Fei Tan

机构 * State Key Laboratory for Novel Software Technology, Nanjing University（南京大学新型软件技术国家重点实验室）； School of Artificial Intelligence, Nanjing University（南京大学人工智能学院）； University of Chicago（芝加哥大学）； The Chinese University of Hong Kong（香港中文大学）； East China Normal University（华东师范大学）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2505.17017 2025-06-11 cs.CV cs.AI cs.CL cs.LG 82%

Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO

Chengzhuo Tong, Ziyu Guo, Renrui Zhang, Wenyu Shan, Xinyu Wei, Zhenghao Xing, Hongsheng Li, Pheng-Ann Heng

机构 * CUHK（香港中文大学）； MiuLar Lab（MiuLar 实验室）； MMLab ； Peking University（北京大学）； Shanghai AI Lab（上海人工智能实验室）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Code is released at https://github.com/ZiyuGuo99/Image-Generation-CoT

2506.05260 2025-06-06 cs.CV 82%

LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs

Xiaodong Wang, Jinfa Huang, Li Yuan, Peixi Peng

机构 * Peking University（北京大学）； Peng Cheng Laboratory（鹏城实验室）

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

Comments Code: https://github.com/Wang-Xiaodong1899/LeanPO

2501.04568 2025-05-20 cs.CV cs.AI cs.CL cs.LG 82%

Feedback-Driven Vision-Language Alignment with Minimal Human Supervision

Giorgio Giannone, Ruoteng Li, Qianli Feng, Evgeny Perevodchikov, Rui Chen, Aleix Martinez

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint

1. 偏好对齐 3229 篇

ACPO: Counteracting Likelihood Displacement in Vision-Language Alignment with Asymmetric Constraints

CausalRM: Causal-Theoretic Reward Modeling for RLHF from Observational User Feedbacks

TARo: Token-level Adaptive Routing for LLM Test-time Alignment

TherapyGym: Evaluating and Aligning Clinical Fidelity and Safety in Therapy Chatbots

UrbanAlign: Post-hoc Semantic Calibration for VLM-Human Preference Alignment

Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization

References Improve LLM Alignment in Non-Verifiable Domains

Less is More: Improving LLM Alignment via Preference Data Selection

MM-SCALE: Grounded Multimodal Moral Reasoning via Scalar Judgment and Listwise Alignment

MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models

VISPA: Pluralistic Alignment via Automatic Value Selection and Activation

MixDPO: Modeling Preference Strength for Pluralistic Alignment

Guiding Cross-Modal Representations with MLLM Priors via Preference Alignment

Align$^3$GR: Unified Multi-Level Alignment for LLM-based Generative Recommendation

PITA: Preference-Guided Inference-Time Alignment for LLM Post-Training

PhysCorr: Dual-Reward DPO for Physics-Constrained Text-to-Video Generation with Automated Preference Selection

Deployable Vision-driven UAV River Navigation via Human-in-the-loop Preference Alignment

Languages are Modalities: Cross-Lingual Alignment via Encoder Injection

ComPO: Preference Alignment via Comparison Oracles

Reward Model Overoptimisation in Iterated RLHF

Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization

Autoregressive End-to-End Planning with Time-Invariant Spatial Alignment and Multi-Objective Policy Refinement

Aligning LLMs on a Budget: Inference-Time Alignment with Heuristic Reward Models

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

Implicit Reward as the Bridge: A Unified View of SFT and DPO Connections

AVC-DPO: Aligned Video Captioning via Direct Preference Optimization

daDPO: Distribution-Aware DPO for Distilling Conversational Abilities

Delving into RL for Image Generation with CoT: A Study on DPO vs. GRPO

LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs

Feedback-Driven Vision-Language Alignment with Minimal Human Supervision