大模型对齐与安全 - arXivDaily 专题

2508.02618 2026-01-14 cs.CL 70%

Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillation

通过交互蒸馏缓解判别奖励建模中的注意力黑客问题

Jianxiang Zang

机构 * College of Computer Science and Artificial Intelligence, Fudan University（计算机科学与人工智能学院，复旦大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL

AI总结本文提出交互蒸馏方法，通过优化注意力机制提升判别奖励建模的稳定性与通用性，缓解注意力黑客问题。

2511.16202 2026-01-06 cs.AI 70%

Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learning

多智能体协作奖励设计以增强强化学习中的推理

Pei Yang, Ke Zhang, Ji Wang, Xiao Chen, Yuxin Tang, Eric Yang, Lynn Ai, Bill Shi

机构 * Gradient ； Waseda University（早稻田大学）； Columbia University（哥伦比亚大学）； Hong Kong Polytechnic University（香港理工大学）； Rice University（莱斯大学）

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.AI

AI总结多智能体协作奖励模型通过分解偏好评估和引入集中化聚合器，提升强化学习中的鲁棒性和可解释性，同时提供透明的奖励建模方法。

2506.21656 2026-01-06 cs.CV cs.CL 70%

Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMs

细粒度偏好优化提升视觉语言模型的空间推理能力

Yifan Shen, Yuanzhe Liu, Jingyuan Zhu, Xu Cao, Xiaofeng Zhang, Yixiao He, Wenming Ye, James Matthew Rehg, Ismini Lourentzou

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

AI总结细粒度偏好优化方法SpatialReasoner-R1通过改进空间推理能力，在空间推理任务中取得显著性能提升。

2312.14925 2025-12-30 cs.LG 70%

A Survey of Reinforcement Learning from Human Feedback

从人类反馈强化学习的综述

Timo Kaufmann, Paul Weng, Viktor Bengs, Eyke Hüllermeier

机构 * LMU Munich（慕尼黑大学）； MCML Munich（慕尼黑马克斯·普朗克研究所）； DFKI（德国人工智能研究中心）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

AI总结本文综述了从人类反馈强化学习的基本原理、核心方法及在多个领域中的应用与贡献。

Comments Published version (TMLR): https://openreview.net/pdf?id=f7OkIurx4b

Journal ref Transactions on Machine Learning Research, 2025

2506.11712 2025-12-23 cs.AI 70%

Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization

通过理论一致的对称多模态偏好优化缓解幻觉

Wenqi Liu, Xuemeng Song, Jiaxi Li, Yinwei Wei, Na Zheng, Jianhua Yin, Liqiang Nie

机构 * Shandong University（山东大学）； Southern University of Science and Technology（南方科技大学）； University of Georgia（佐治亚大学）； National University of Singapore（新加坡国立大学）； Harbin Institute of Technology (Shenzhen)（哈尔滨工业大学（深圳））

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

AI总结 SymMPO通过理论一致的对称多模态偏好优化方法，有效缓解多模态大语言模型中的幻觉问题。

Comments NeurIPS 2025

2512.17213 2025-12-22 cs.CV cs.LG 70%

CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency

CheXPO-v2：基于知识图谱一致性的胸片视觉-语言模型偏好优化

Xiao Liang, Yuxuan An, Di Wang, Jiawei Hu, Zhicheng Jiao, Bin Jing, Quan Wang

机构 * School of Computer Science and Technology, Xidian University（西安电子科技大学计算机科学与技术学院）； Warren Alpert Medical School of Brown University（布朗大学沃伦·阿尔珀特医学学院）； School of Biomedical Engineering, Capital Medical University（首都医科大学生物医学工程学院）

专题命中偏好对齐：alignment(abstract)；safety(abstract)；分类 cs.LG

AI总结 CheXPO-v2通过知识图谱一致性奖励机制，提升胸片VLMs的临床可靠性与数据效率。

2505.11711 2025-12-19 cs.LG 70%

Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

强化学习在大型语言模型中微调小型子网络

Sagnik Mukherjee, Lifan Yuan, Dilek Hakkani-Tur, Hao Peng

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.LG

AI总结强化学习在大型语言模型中通过微调小型子网络显著提升性能，且该子网络更新稀疏性源于数据分布和正则化技术的影响。

Comments NeuRIPS 2025

2509.23188 2025-12-16 cs.CL 70%

Diagnose, Localize, Align: A Full-Stack Framework for Reliable LLM Multi-Agent Systems under Instruction Conflicts

诊断、定位、对齐：一种用于在指令冲突下可靠LLM多智能体系统的全栈框架

Guancheng Wan, Leixin Sun, Longxu Dou, Zitong Shi, Fang Wu, Eric Hanchen Jiang, Wenke Huang, Guibin Zhang, Hejia Geng, Xiangru Tang, Zhenfei Yin, Yizhou Sun, Wei Wang

机构 * University of California, Los Angeles（加州大学洛杉矶分校）； Sea AI Lab（Sea AI 实验室）； Stanford University（斯坦福大学）； University of Oxford（牛津大学）； Yale University（耶鲁大学）； NTU（南洋理工大学）； NUS（新加坡国立大学）； Boston University（波士顿大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

AI总结本文提出了一种全栈框架，通过诊断、定位和对齐三个阶段提升LLM多智能体系统在指令冲突下的可靠性。

Comments Upon further review, we realized that the version submitted to arXiv was not the final draft and omits crucial results and discussion. To avoid confusion and ensure the integrity of the record, we request withdrawal and will resubmit once the complete work is ready

2512.06920 2025-12-09 cs.LG 70%

Parent-Guided Semantic Reward Model (PGSRM): Embedding-Based Reward Functions for Reinforcement Learning of Transformer Language Models

基于嵌入的语义奖励模型（PGSRM）：用于变换器语言模型强化学习的奖励函数

Alexandr Plashchinsky

机构 * VECTOR Labs（VECTOR实验室）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

AI总结 PGSRM通过嵌入相似度生成语义奖励，为变换器语言模型强化学习提供无需人工标注的替代方案。

2507.20109 2025-12-09 cs.SE cs.AI 70%

Learning to Align Human Code Preferences

学习对齐人类代码偏好

Xin Yin, Chao Ni, Xiaohu Yang

机构 * Zhejiang University（浙江大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

AI总结本文提出自适应偏好优化（APO）方法，通过动态整合SFT和DPO，提升模型在不同代码偏好场景下的对齐性能。

2506.01790 2025-12-08 cs.LG cs.CR 70%

IF-GUIDE: Influence Function-Guided Detoxification of LLMs

IF-GUIDE：影响函数引导的LLM去毒化

Zachary Coalson, Juhan Bae, Nicholas Carlini, Sanghyun Hong

机构 * Oregon State University（俄勒冈州立大学）； University of Toronto（多伦多大学）； Anthropic

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.LG

AI总结 IF-GUIDE通过影响函数主动识别并抑制训练数据中的有害标记，有效减少大语言模型的显性和隐性毒性。

Comments Accepted at NeurIPS 2025 [Poster]

2512.03208 2025-12-04 stat.ML cs.LG 70%

Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback

大语言模型奖励学习中异质人类反馈的不确定性量化

Pangpang Liu, Junwei Lu, Will Wei Sun

机构 * Department of Biostatistics, Yale University（耶鲁大学生物统计学系）； Department of Biostatistics, Harvard University（哈佛大学生物统计学系）； Department of Quantitative Methods, Purdue University（普渡大学定量方法系）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

AI总结本文提出了一种异质偏好框架，通过双凸优化解决大语言模型奖励学习中的不确定性量化问题，并通过理论保证和实验证明其有效性。

2511.06942 2025-12-03 cs.CL cs.CR 70%

HLPD: Aligning LLMs to Human Language Preference for Machine-Revised Text Detection

HLPD: 通过人类语言偏好对齐大语言模型以检测机器修订文本

Fangqi Dai, Xingjian Jiang, Zizhuang Deng

专题命中偏好对齐：alignment(abstract)；trustworthy(abstract)；分类 cs.CL

AI总结 HLPD通过人类语言偏好优化提升机器修订文本检测性能，实现对GPT系列模型和先进大语言模型生成文本的高效识别。

Comments 20 pages, 10 figures, accepted by AAAI'26

2406.17636 2025-12-03 cs.CV cs.AI 70%

Aligning Diffusion Models with Noise-Conditioned Perception

对扩散模型进行噪声条件感知对齐

Alexander Gambashidze, Anton Kulikov, Yuriy Sosnin, Ilya Makarov

机构 * Artificial Intelligence Research Institute (AIRI)（人工智能研究 institute（AIRI））； Skolkovo Institute of Science and Technology（斯克尔科沃科学与技术研究所）； HSE University（俄罗斯高等经济学院）； Research Center for Trusted Artificial Intelligence, ISP RAS（可信人工智能研究中心，信息与通信技术研究院）； ISP RAS（信息与通信技术研究院）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

AI总结本研究提出在扩散模型的U-Net嵌入空间中使用感知目标，以提升人类偏好对齐的效率和质量，并减少计算成本。

2511.23436 2025-12-01 cs.AI 70%

Towards Continuous Intelligence Growth: Self-Training, Continual Learning, and Dual-Scale Memory in SuperIntelliAgent

迈向持续智能增长：在SuperIntelliAgent中实现自我训练、持续学习和双尺度记忆

Jianzhe Lin, Zeyu Pan, Yun Zhu, Ruiqi Song, Jining Yang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

AI总结 SuperIntelliAgent通过自我训练和双尺度记忆实现持续智能增长，利用可训练学习者与推理验证者配对，提升偏好对齐和学习效果。

Comments 15 pages, 4 figures

2511.22138 2025-12-01 cs.LG 70%

TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devices

TinyLLM: 小型语言模型在边缘设备上用于代理任务的评估与优化

Mohd Ariful Haque, Fahad Rahman, Kishor Datta Gupta, Khalil Shujaee, Roy George

机构 * Department of Cyber-Physical Systems, Clark Atlanta University, USA（计算机物理系统系，Clark Atlanta大学，美国）； Department of Computer Science and Engineering, United International University, Bangladesh（计算机科学与工程系，联合国际大学，孟加拉国）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.LG

AI总结 TinyLLM研究小型语言模型在边缘设备上执行代理任务的优化方法，通过混合策略提升准确性与效率，验证了中等规模模型在多轮任务中的优势。

Comments 8 pages, 3 figures, 4 tables

2507.04340 2025-12-01 cs.LG cs.HC 70%

Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback

基于交互式组间比较的强化学习从人类反馈中学习

Jan Kompatscher, Danqing Shi, Giovanna Varni, Tino Weinkauf, Antti Oulasvirta

机构 * Aalto University（阿尔托大学）； University of Cambridge（剑桥大学）； University of Trento（特伦特大学）； KTH Royal Institute of Technology（皇家理工学院）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

AI总结本文提出交互式组间比较方法，通过可视化和主动学习提升强化学习从人类反馈中学习的奖励和策略效果。

Comments 10 pages, 8 figures in proceedings of Computer Graphics Forum

2508.08746 2025-11-26 cs.LG 70%

Interpretable Reward Model via Sparse Autoencoder

通过稀疏自编码器的可解释奖励模型

Shuyi Zhang, Wei Shi, Sihang Li, Jiayi Liao, Hengxing Cai, Xiang Wang

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

AI总结 SARM通过整合预训练稀疏自编码器，提升奖励模型的可解释性和对齐性能。

Comments AAAI 2026 Oral

2411.17265 2025-11-25 cs.CL cs.CV 70%

Systematic Reward Gap Optimization for Mitigating VLM Hallucinations

系统性奖励缺口优化以缓解视觉语言模型的幻觉

Lehan He, Zeren Chen, Zhelun Shi, Tianyu Yu, Jing Shao, Lu Sheng

机构 * School of Software, Beihang University（北京航空航天大学软件学院）； Shanghai Innovation Institute（上海创新研究院）； Shanghai AI Laboratory（上海人工智能实验室）； Tsinghua University（清华大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

AI总结 TPR通过主题级偏好重写系统优化奖励缺口配置，显著减少VLM幻觉并提升对齐效果。

Comments 34 pages, 12 figures, Accepted by NeurIPS 2025

2511.14221 2025-11-19 cs.IR cs.AI 70%

LLM-Aligned Geographic Item Tokenization for Local-Life Recommendation

Hao Jiang, Guoquan Wang, Donglin Zhou, Sheng Yu, Yang Zeng, Wencong Zeng, Kun Gai, Guorui Zhou

机构 * Kuaishou Technology（快手科技）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

2511.13016 2025-11-18 cs.LG 70%

The Good, The Bad, and The Hybrid: A Reward Structure Showdown in Reasoning Models Training

Subramanyam Sahoo

机构 * Berkeley AI Safety Initiative (BASIS) UC Berkeley（伯克利人工智能安全计划（BASIS）加州大学伯克利分校）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

Comments Paper accepted to the 2nd Workshop on Aligning Reinforcement Learning Experimentalists and Theorists (ARLET 2025) at NeurIPS; the paper consists of 14 pages (including the appendix) and contains 3 figures

2506.19248 2025-11-06 cs.LG 70%

Inference-Time Reward Hacking in Large Language Models

Hadi Khalaf, Claudio Mayrink Verdun, Alex Oesterling, Himabindu Lakkaraju, Flavio du Pin Calmon

机构 * Harvard University（哈佛大学）

专题命中偏好对齐：alignment(abstract)；safety(abstract)；分类 cs.LG

Comments Accepted to NeurIPS 2025 (Spotlight Paper)

2510.23590 2025-10-28 cs.LG 70%

Lightweight Robust Direct Preference Optimization

Cheol Woo Kim, Shresth Verma, Mauricio Tec, Milind Tambe

机构 * School of Engineering and Applied Sciences, Harvard University（哈佛大学工程与应用科学学院）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.LG

Comments arXiv admin note: substantial text overlap with arXiv:2509.02709

2510.22954 2025-10-28 cs.CL 70%

Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)

Liwei Jiang, Yuanjun Chai, Margaret Li, Mickel Liu, Raymond Fok, Nouha Dziri, Yulia Tsvetkov, Maarten Sap, Alon Albalak, Yejin Choi

专题命中偏好对齐：safety(abstract)；AI safety(abstract)；分类 cs.CL

Comments NeurIPS 2025 D&B Paper (Oral); Camera-Ready Version

2510.22084 2025-10-28 cs.CL 70%

Compositional Bias Control in Large Language Models: Preference Learning Fails, Supervision Succeeds

Atij Mahesh

机构 * University of California, Los Angeles（加州大学洛杉矶分校）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments 20 pages

2510.20369 2025-10-24 cs.LG 70%

Ask a Strong LLM Judge when Your Reward Model is Uncertain

Zhenghao Xu, Qin Lu, Qingru Zhang, Liang Qiu, Ilgee Hong, Changlong Yu, Wenlin Yao, Yao Liu, Haoming Jiang, Lihong Li, Hyokun Yun, Tuo Zhao

机构 * Georgia Institute of Technology（佐治亚理工学院）； Amazon（亚马逊）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

Comments NeurIPS 2025, 18 pages

2509.10935 2025-10-22 cs.CL 70%

Introducing Spotlight: A Novel Approach for Generating Captivating Key Information from Documents

Ankan Mullick, Sombit Bose, Rounak Saha, Ayan Kumar Bhowmick, Aditya Vempaty, Prasenjit Dey, Ravi Kokku, Pawan Goyal, Niloy Ganguly

机构 * IIT Kharagpur（印度克达尔普大学）； Emergence AI

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments Paper accepted in EMNLP 2025 Main Conference (Full Paper)

Journal ref EMNLP 2025 Main Conference (Full Paper)

2510.12041 2025-10-16 cs.CL 70%

Improving Text-to-Image Generation with Input-Side Inference-Time Scaling

Ruibo Chen, Jiacheng Pan, Heng Huang, Zhenheng Yang

机构 * TikTok ； University of Maryland, College Park（马里兰大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

2510.06652 2025-10-09 cs.CL 70%

Aligning Large Language Models via Fully Self-Synthetic Data

Shangjian Yin, Zhepei Wei, Xinyu Zhu, Wei-Lin Chen, Yu Meng

机构 * University of California, Riverside（加州大学河滨分校）； University of Virginia（弗吉尼亚大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL

2410.12491 2025-10-07 cs.CL 70%

Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning

Jared Joselowitz, Ritam Majumdar, Arjun Jagota, Matthieu Bou, Nyal Patel, Satyapriya Krishna, Sonali Parbhoo

机构 * Imperial College London（伦敦帝国学院）； Harvard University（哈佛大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL

Comments Published as a conference paper at COLM 2025

1. 偏好对齐 3238 篇

Alleviating Attention Hacking in Discriminative Reward Modeling through Interaction Distillation

Multi-Agent Collaborative Reward Design for Enhancing Reasoning in Reinforcement Learning

Fine-Grained Preference Optimization Improves Spatial Reasoning in VLMs

A Survey of Reinforcement Learning from Human Feedback

Mitigating Hallucination Through Theory-Consistent Symmetric Multimodal Preference Optimization

CheXPO-v2: Preference Optimization for Chest X-ray VLMs with Knowledge Graph Consistency

Reinforcement Learning Finetunes Small Subnetworks in Large Language Models

Diagnose, Localize, Align: A Full-Stack Framework for Reliable LLM Multi-Agent Systems under Instruction Conflicts

Parent-Guided Semantic Reward Model (PGSRM): Embedding-Based Reward Functions for Reinforcement Learning of Transformer Language Models

Learning to Align Human Code Preferences

IF-GUIDE: Influence Function-Guided Detoxification of LLMs

Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback

HLPD: Aligning LLMs to Human Language Preference for Machine-Revised Text Detection

Aligning Diffusion Models with Noise-Conditioned Perception

Towards Continuous Intelligence Growth: Self-Training, Continual Learning, and Dual-Scale Memory in SuperIntelliAgent

TinyLLM: Evaluation and Optimization of Small Language Models for Agentic Tasks on Edge Devices

Interactive Groupwise Comparison for Reinforcement Learning from Human Feedback

Interpretable Reward Model via Sparse Autoencoder

Systematic Reward Gap Optimization for Mitigating VLM Hallucinations

LLM-Aligned Geographic Item Tokenization for Local-Life Recommendation

The Good, The Bad, and The Hybrid: A Reward Structure Showdown in Reasoning Models Training

Inference-Time Reward Hacking in Large Language Models

Lightweight Robust Direct Preference Optimization

Artificial Hivemind: The Open-Ended Homogeneity of Language Models (and Beyond)

Compositional Bias Control in Large Language Models: Preference Learning Fails, Supervision Succeeds

Ask a Strong LLM Judge when Your Reward Model is Uncertain

Introducing Spotlight: A Novel Approach for Generating Captivating Key Information from Documents

Improving Text-to-Image Generation with Input-Side Inference-Time Scaling

Aligning Large Language Models via Fully Self-Synthetic Data

Insights from the Inverse: Reconstructing LLM Training Goals Through Inverse Reinforcement Learning