大模型对齐与安全 - arXivDaily 专题

2509.23715 2025-09-30 cs.CL cs.LG 62%

Do LLMs Understand Romanian Driving Laws? A Study on Multimodal and Fine-Tuned Question Answering

Eduard Barbu, Adrian Marius Dumitran

机构 * Faculty of Mathematics and Informatics, University of Bucharest（数学与信息学系，布加勒斯特大学）

专题命中偏好对齐：safety(abstract)；分类 cs.CL、cs.LG

Comments Accepted@ CONSILR 2025 Bucharest Romania 9-10 October

2509.23368 2025-09-30 cs.CL cs.AI 62%

MedCritical: Enhancing Medical Reasoning in Small Language Models via Self-Collaborative Correction

Xinchun Su, Chunxu Luo, Yixuan Li, Weidong Yang, Lipeng Ma

机构 * School of Computer Science, Fudan University, Shanghai, China（复旦大学计算机学院）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI

2508.15845 2025-09-30 cs.CL cs.AI 62%

Coarse-to-Fine Personalized LLM Impressions for Streamlined Radiology Reports

Chengbo Sun, Hui Yi Leong, Lei Li

机构 * University of Chicago（芝加哥大学）； University of Washington（华盛顿大学）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI

2505.12716 2025-09-29 cs.CL cs.AI 62%

Shadow-FT: Tuning Instruct Model via Training on Paired Base Model

Taiqiang Wu, Runming Yang, Jiayi Li, Pengfei Hu, Yik-Chung Wu, Ngai Wong, Yujiu Yang

机构 * The University of Hong Kong（香港大学）； Tsinghua University（清华大学）； Tencent（腾讯）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI

Comments 24 pages, 12 tables, 8 figures. Previous name: Shadow-FT: Tuning Instruct via Base

2406.07780 2025-09-29 cs.LG cs.CL 62%

A Critical Look At Tokenwise Reward-Guided Text Generation

Ahmad Rashid, Ruotian Wu, Julia Grosse, Agustinus Kristiadi, Pascal Poupart

机构 * University of Waterloo（滑铁卢大学）； Vector Institute（向量研究所）； University of Tübingen（图宾根大学）； Tübingen AI Center（图宾根人工智能中心）； Western University（西方大学）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.LG

Comments Work accepted at COLM 2025

2507.16663 2025-09-26 cs.CL cs.AI 62%

Turning Internal Gap into Self-Improvement: Promoting the Generation-Understanding Unification in MLLMs

Yujin Han, Hao Chen, Andi Han, Zhiheng Wang, Xinyu Liu, Yingya Zhang, Shiwei Zhang, Difan Zou

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI

Comments 31 pages, 16 figures, 12 tables

2509.19336 2025-09-25 cs.CL cs.AI 62%

Cognitive-Level Adaptive Generation via Capability-Aware Retrieval and Style Adaptation

Qingsong Wang, Tao Wu, Wang Lin, Yueying Feng, Gongsheng Yuan, Chang Yao, Jingyuan Chen

机构 * Zhejiang University（浙江大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI

Comments Accepted to Findings of EMNLP 2026

2509.19265 2025-09-24 cs.AI cs.CL 62%

Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World

Saeed Almheiri, Rania Hossam, Mena Attia, Chenxi Wang, Preslav Nakov, Timothy Baldwin, Fajri Koto

机构 * Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2025 - Findings

2509.18316 2025-09-24 cs.CL cs.AI 62%

Brittleness and Promise: Knowledge Graph Based Reward Modeling for Diagnostic Reasoning

Saksham Khatwani, He Cheng, Majid Afshar, Dmitriy Dligach, Yanjun Gao

机构 * University of Colorado Boulder（科罗拉多大学博尔德分校）； University of Colorado Anschutz（科罗拉多大学安舒茨分校）； University of Wisconsin - Madison（威斯康星大学麦迪逊分校）； Loyola University Chicago（芝加哥洛约拉大学）

专题命中偏好对齐：trustworthy(abstract)；分类 cs.CL、cs.AI

2506.04039 2025-09-23 cs.CV cs.AI cs.CL 62%

Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization

Jiulong Wu, Zhengliang Shi, Shuaiqiang Wang, Jizhou Huang, Dawei Yin, Lingyong Yan, Min Cao, Min Zhang

机构 * Soochow University（苏州大学）； Baidu Inc.（百度公司）； Shandong University（山东大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI

Comments This paper is accepted by EMNLP2025

2509.16475 2025-09-23 cs.LG cs.CL 62%

Towards Universal Debiasing for Language Models-based Tabular Data Generation

Tianchun Li, Tianci Liu, Xingchen Wang, Rongzhe Wei, Pan Li, Lu Su, Jing Gao

机构 * Purdue University（普渡大学）； Georgia Institute of Technology（佐治亚理工学院）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.LG

Comments EMNLP 2025 Findings

2505.14442 2025-09-22 cs.CL cs.AI 62%

Creative Preference Optimization

Mete Ismayilzada, Antonio Laverghetta, Simone A. Luchini, Reet Patel, Antoine Bosselut, Lonneke van der Plas, Roger Beaty

机构 * EPFL（苏黎世联邦理工学院）； Università della Svizzera Italiana（瑞士联邦理工学院）； Wesleyan University（韦斯利安大学）； Pennsylvania State University（宾夕法尼亚州立大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2025 Findings

2503.12123 2025-09-22 cs.CL cs.AI 62%

MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modeling

Zhaopeng Feng, Jiahan Ren, Jiayuan Su, Jiamei Zheng, Hongwei Wang, Zuozhu Liu

机构 * Zhejiang University（浙江大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2025 Findings. Project page:https://sabijun.github.io/MT_RewardTreePage

2412.11006 2025-09-22 cs.LG cs.CL 62%

Entropy-Regularized Process Reward Model

Hanning Zhang, Pengcheng Wang, Shizhe Diao, Yong Lin, Rui Pan, Hanze Dong, Dylan Zhang, Pavlo Molchanov, Tong Zhang

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）； University of Toronto（多伦多大学）； NVIDIA（英伟达）； Princeton University（普林斯顿大学）； Salesforce Research（Salesforce研究）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.LG

Comments Upate TMLR version

2503.01658 2025-09-18 cs.LG cs.AI cs.IR 62%

CoPL: Collaborative Preference Learning for Personalizing LLMs

Youngbin Choi, Seunghyuk Cho, Minjong Lee, MoonJeong Park, Yesong Ko, Jungseul Ok, Dongwoo Kim

机构 * Graduate School of Artificial Intelligence, POSTECH（POSTECH人工智能研究生院）； Department of Computer Science and Engineering, POSTECH（POSTECH计算机科学与工程系）

专题命中偏好对齐：alignment(abstract)；分类 cs.AI、cs.LG

Comments 19pages, 13 figures, 11 tables

2503.12301 2025-09-16 cs.LG cs.CL 62%

One Goal, Many Challenges: Robust Preference Optimization Amid Content-Aware and Multi-Source Noise

Amirabbas Afzali, Amirhossein Afsharrad, Seyed Shahabeddin Mousavi, Sanjay Lall

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.LG

2503.01062 2025-09-16 cs.LG cs.AI 62%

Offline RLAIF: Piloting VLM Feedback for RL via SFO

Jacob Beck

机构 * Jacob Beck 1

专题命中偏好对齐：RLHF(abstract)；分类 cs.AI、cs.LG

Comments Code is provided at https://github.com/jacooba/OfflineRLAIF

Journal ref Published at The RLC 2025 Workshop on Reinforcement Learning Beyond Rewards: Ingredients for Developing Generalist Agents

2411.08302 2025-09-12 cs.CL cs.AI 62%

RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution

Jiahui Li, Lin Li, Tai-wei Chang, Kun Kuang, Long Chen, Jun Zhou, Cheng Yang

机构 * Zhejiang University（浙江大学）； Ant Group（蚂蚁集团）； The Hong Kong University of Science and Technology（香港科技大学）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI

2508.19576 2025-09-09 cs.AI cs.LG 62%

ReST-RL: Achieving Accurate Code Reasoning of LLMs with Optimized Self-Training and Decoding

Sining Zhoubian, Dan Zhang, Jie Tang

机构 * The Knowledge Engineering Group (KEG), Tsinghua University（清华大学知识工程小组（KEG）、清华大学）

专题命中偏好对齐：DPO(abstract)；分类 cs.AI、cs.LG

Comments 21 pages, 4 figures

2506.07963 2025-09-09 cs.AI cs.CL cs.CV 62%

SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards

Jixiang Hong, Yiran Zhang, Guanzhong Wang, Yi Liu, Ji-Rong Wen, Rui Yan

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）； School of Computer Science（计算机科学学院）； Baidu Inc.（百度公司）； School of Computer Science, Wuhan University（武汉大学计算机学院）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI

2509.05605 2025-09-09 cs.CL cs.AI 62%

Icon$^{2}$: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation

Qiyuan Chen, Hongsen Huang, Qian Shao, Jiahe Chen, Jintai Chen, Hongxia Xu, Renjie Hua, Ren Chuan, Jian Wu

机构 * Zhejiang University（浙江大学）； Soochow Securities Co.,Ltd.（苏州证券有限公司）； HKUST(GZ)（香港科技大学（广州））； Nanjing University（南京大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2025 Main

2508.17637 2025-08-26 cs.CL cs.AI 62%

Weights-Rotated Preference Optimization for Large Language Models

Chenxu Yang, Ruipeng Jia, Mingyu Zheng, Naibin Gu, Zheng Lin, Siyuan Chen, Weichong Yin, Hua Wu, Weiping Wang

机构 * Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所）； School of Cyber Security, University of Chinese Academy of Sciences（中国科学院大学网络安全学院）； Baidu Inc.（百度公司）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2025

2508.15652 2025-08-26 cs.AI cs.IT cs.LG cs.MA math.IT 62%

Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learning

Ardian Selmonaj, Miroslav Strupl, Oleg Szehr, Alessandro Antonucci

机构 * Istituto Dalle Molle di Studi sull’Intelligenza Artificiale (IDSIA), USI-SUPSI（日内瓦人工智能研究所（IDSIA）、USI-SUPSI）

专题命中偏好对齐：alignment(abstract)；分类 cs.AI、cs.LG

Comments European Conference on Artificial Intelligence (ECAI) 2025

2508.15790 2025-08-25 cs.CL cs.AI 62%

KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integration

Nan Wang, Yongqi Fan, yansha zhu, ZongYu Wang, Xuezhi Cao, Xinyan He, Haiyun Jiang, Tong Ruan, Jingping Liu

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI

2502.01819 2025-08-25 cs.LG cs.AI math.OC 62%

Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning

Hanyang Zhao, Haoxian Chen, Ji Zhang, David D. Yao, Wenpin Tang

机构 * StonyBrook（石溪大学）

专题命中偏好对齐：RLHF(abstract)；分类 cs.AI、cs.LG

Comments arXiv admin note: text overlap with arXiv:2409.08400

2508.15449 2025-08-22 cs.LG cs.AI 62%

Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection

Chengcan Wu, Zeming Wei, Huanran Chen, Yinpeng Dong, Meng Sun

机构 * Peking University（北京大学）； Tsinghua University（清华大学）

专题命中偏好对齐：safety(abstract)；分类 cs.AI、cs.LG

Comments 10 pages, 9 figures, Under review as a full paper at AAAI 2026. A preliminary version is under review at the NeurIPS 2025 Workshop on Reliable ML from Unreliable Data

2504.16438 2025-08-20 cs.LG cs.AI cs.CR cs.DC 62%

POPri: Private Federated Learning using Preference-Optimized Synthetic Data

Charlie Hou, Mei-Yu Wang, Yige Zhu, Daniel Lazar, Giulia Fanti

机构 * Pittsburgh Supercomputing Center, Pittsburgh, USA（匹兹堡超级计算中心）； Department of ECE, Carnegie Mellon University, Pittsburgh, PA（电子工程系，卡内基梅隆大学）

专题命中偏好对齐：DPO(abstract)；分类 cs.AI、cs.LG

Comments ICML 2025 camera-ready

2508.13250 2025-08-20 cs.AI cs.CL cs.IR 62%

Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized Information

Zeyu Zhang, Yang Zhang, Haoran Tan, Rui Li, Xu Chen

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）； National University of Singapore（新加坡国立大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 13 figures, 3 tables

2508.13189 2025-08-20 stat.ML cs.AI cs.LG 62%

Preference Models assume Proportional Hazards of Utilities

Chirag Nagpal

机构 * Meta Superintelligence Labs (MSL)（Meta超智能实验室）

专题命中偏好对齐：alignment(abstract)；分类 cs.AI、cs.LG

2508.07970 2025-08-19 cs.LG cs.AI 62%

WeChat-YATT: A Scalable, Simple, Efficient, and Production Ready Training Library

Junyu Wu, Weiming Chang, Xiaotao Liu, Guanyou He, Tingfeng Xian, Haoqiang Hong, Boqi Chen, Hongtao Tian, Tao Yang, Yunsheng Shi, Feng Lin, Ting Yao, Jiatao Xu

机构 * Tencent（腾讯）

专题命中偏好对齐：RLHF(abstract)；分类 cs.AI、cs.LG

Comments arXiv admin note: substantial text overlap with arXiv:2507.22789

1. 偏好对齐 3243 篇

Do LLMs Understand Romanian Driving Laws? A Study on Multimodal and Fine-Tuned Question Answering

MedCritical: Enhancing Medical Reasoning in Small Language Models via Self-Collaborative Correction

Coarse-to-Fine Personalized LLM Impressions for Streamlined Radiology Reports

Shadow-FT: Tuning Instruct Model via Training on Paired Base Model

A Critical Look At Tokenwise Reward-Guided Text Generation

Turning Internal Gap into Self-Improvement: Promoting the Generation-Understanding Unification in MLLMs

Cognitive-Level Adaptive Generation via Capability-Aware Retrieval and Style Adaptation

Cross-Cultural Transfer of Commonsense Reasoning in LLMs: Evidence from the Arab World

Brittleness and Promise: Knowledge Graph Based Reward Modeling for Diagnostic Reasoning

Mitigating Hallucinations in Large Vision-Language Models via Entity-Centric Multimodal Preference Optimization

Towards Universal Debiasing for Language Models-based Tabular Data Generation

Creative Preference Optimization

MT-RewardTree: A Comprehensive Framework for Advancing LLM-Based Machine Translation via Reward Modeling

Entropy-Regularized Process Reward Model

CoPL: Collaborative Preference Learning for Personalizing LLMs

One Goal, Many Challenges: Robust Preference Optimization Amid Content-Aware and Multi-Source Noise

Offline RLAIF: Piloting VLM Feedback for RL via SFO

RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution

ReST-RL: Achieving Accurate Code Reasoning of LLMs with Optimized Self-Training and Decoding

SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards

Icon$^{2}$: Aligning Large Language Models Using Self-Synthetic Preference Data via Inherent Regulation

Weights-Rotated Preference Optimization for Large Language Models

Understanding Action Effects through Instrumental Empowerment in Multi-Agent Reinforcement Learning

KG-o1: Enhancing Multi-hop Question Answering in Large Language Models via Knowledge Graph Integration

Score as Action: Fine-Tuning Diffusion Generative Models by Continuous-time Reinforcement Learning

Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection

POPri: Private Federated Learning using Preference-Optimized Synthetic Data

Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized Information

Preference Models assume Proportional Hazards of Utilities

WeChat-YATT: A Scalable, Simple, Efficient, and Production Ready Training Library