大模型对齐与安全 - arXivDaily 专题

2509.06759 2025-09-09 cs.LG cs.AI 73%

Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization

Thanh Thi Nguyen, Campbell Wilson, Janis Dalins

机构 * AiLECS Lab, Monash University Melbourne, Australia（墨尔本大学AiLECS实验室，澳大利亚）； AiLECS Lab, Monash University, Australia ICMEC Australia, Sydney, Australia（墨尔本大学AiLECS实验室，澳大利亚 ICMEC澳大利亚，悉尼，澳大利亚）

专题命中偏好对齐：DPO(abstract)；safety(abstract)；分类 cs.AI、cs.LG

Comments Accepted for publication in the Proceedings of the 8th International Conference on Algorithms, Computing and Artificial Intelligence (ACAI 2025)

2508.21476 2025-09-01 cs.CL cs.AI 73%

Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards

Xiaolong Wei, Bo Lu, Xingyu Zhang, Zhejun Zhao, Dongdong Shen, Long Xia, Dawei Yin

机构 * Beihang University（北航）； Baidu Inc.（百度公司）； Beijing Jiaotong University（北京交通大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2025 Main

2508.20181 2025-08-29 cs.CV cs.AI cs.CL cs.MM 73%

Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization

Alberto Compagnoni, Davide Caffagni, Nicholas Moratelli, Lorenzo Baraldi, Marcella Cornia, Rita Cucchiara

机构 * University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments BMVC 2025

2508.16741 2025-08-26 cs.LG cs.AI 73%

WST: Weak-to-Strong Knowledge Transfer via Reinforcement Learning

Haosen Ge, Shuo Li, Lianghuan Huang

机构 * Wharton AI & Analytics Initiative（沃顿人工智能与分析倡议）； University of Pennsylvania（宾夕法尼亚大学）； Department of Computer and Information Science（计算机与信息科学系）； Department of Physics and Astronomy（物理学与天文学系）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2507.21391 2025-07-31 cs.CV cs.AI cs.CL 73%

Multimodal LLMs as Customized Reward Models for Text-to-Image Generation

Shijie Zhou, Ruiyi Zhang, Huaisheng Zhu, Branislav Kveton, Yufan Zhou, Jiuxiang Gu, Jian Chen, Changyou Chen

机构 * University at Buffalo（布法罗大学）； Adobe Research（Adobe研究）； Pennsylvania State University（宾夕法尼亚州立大学）

专题命中偏好对齐：alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICCV 2025. Code available at https://github.com/sjz5202/LLaVA-Reward

2507.20181 2025-07-29 cs.CL cs.AI 73%

SGPO: Self-Generated Preference Optimization based on Self-Improver

Hyeonji Lee, Daejin Jo, Seohwan Yun, Sungwoong Kim

机构 * Korea University（韩国大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2506.21560 2025-07-29 cs.CL cs.AI 73%

Reinforcement learning fine-tuning of language model for instruction following and math reasoning

Yifu Han, Geo Zhang

机构 * Stanford University（斯坦福大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2507.18417 2025-07-25 cs.CL cs.LG q-fin.ST q-fin.TR 73%

FinDPO: Financial Sentiment Analysis for Algorithmic Trading through Preference Optimization of LLMs

Giorgos Iacovides, Wuyang Zhou, Danilo Mandic

机构 * Imperial College London（伦敦帝国理工学院）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2411.06208 2025-07-18 cs.CL cs.AI 73%

IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization

Xinghua Zhang, Haiyang Yu, Cheng Fu, Fei Huang, Yongbin Li

机构 * Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments ACL 2025

2507.09973 2025-07-15 cs.CL cs.AI 73%

Tiny Reward Models

Sarah Pan

机构 * Massachusetts Institute of Technology（麻省理工学院）

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI

Comments 2025 ICML Efficient Systems for Foundation Models Workshop

2407.06902 2025-07-04 eess.SP cs.AI cs.HC cs.LG 73%

Learning From Crowdsourced Noisy Labels: A Signal Processing Perspective

Shahana Ibrahim, Panagiotis A. Traganitis, Xiao Fu, Georgios B. Giannakis

机构 * Department of Electrical and Computer Engineering, University of Central Florida（中央佛罗里达大学电气与计算机工程系）； School of Electrical Engineering and Computer Science, Oregon State University（俄勒冈州立大学电气工程与计算机科学学院）； Department of Electrical and Computer Engineering, University of Minnesota（明尼苏达大学电气与计算机工程系）

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2507.01479 2025-07-03 cs.CL cs.AI 73%

Evaluating the Effectiveness of Direct Preference Optimization for Personalizing German Automatic Text Simplifications for Persons with Intellectual Disabilities

Yingqiang Gao, Kaede Johnson, David Froehlich, Luisa Carrer, Sarah Ebling

机构 * University of Zurich（苏黎世大学）； EPFL（瑞士联邦理工学院）； Zurich University of Applied Sciences（苏黎世应用科学大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2405.17618 2025-06-24 cs.LG cs.AI 73%

Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales

Ju-Seung Byun, Andrew Perrault

机构 * Department of Computer Science and Engineering, The Ohio State University（计算机科学与工程系，俄亥俄州立大学）； Department of Computer Science（计算机科学系）； Engineering, The Ohio State University（工程系，俄亥俄州立大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2504.04524 2025-06-19 cs.LG cs.AI 73%

Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoning

Xuerui Su, Shufang Xie, Guoqing Liu, Yingce Xia, Renqian Luo, Peiran Jin, Zhiming Ma, Yue Wang, Zun Wang, Yuting Liu

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments 10pages

2506.13599 2025-06-17 cs.CL cs.AI 73%

CAMS: A CityGPT-Powered Agentic Framework for Urban Human Mobility Simulation

Yuwei Du, Jie Feng, Jian Yuan, Yong Li

机构 * Department of Electronic Engineering, BRNist, Tsinghua University, Beijing, China（电子工程系，BRNist，清华大学，北京，中国）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2506.09096 2025-06-17 cs.LG cs.AI 73%

Intra-Trajectory Consistency for Reward Modeling

Chaoyang Zhou, Shunyu Liu, Zengmao Wang, Di Wang, Rong-Cheng Tu, Bo Du, Dacheng Tao

机构 * Wuhan University（武汉大学）； Nanyang Technological University（南洋理工大学）

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments Under review

2502.13131 2025-06-12 cs.AI cs.CL 73%

Rethinking Diverse Human Preference Learning through Principal Component Analysis

Feng Luo, Rui Yang, Hao Sun, Chunyuan Deng, Jiarui Yao, Jingyan Shen, Huan Zhang, Hanjie Chen

机构 * Rice University（里士大学）； University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）； University of Cambridge（剑桥大学）； Columbia University（哥伦比亚大学）

专题命中偏好对齐：alignment(abstract)；safety(abstract)；分类 cs.CL、cs.AI

Comments 14 pages

2410.20898 2025-06-06 cs.CV cs.AI cs.LG cs.MM 73%

David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training

Weijian Luo, Colin Zhang, Debing Zhang, Zhengyang Geng

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments Revision: paper accepted by the ICML2025 main conference

2410.18881 2025-06-06 cs.CV cs.AI cs.LG 73%

Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences

Weijian Luo

机构 * Peking University（北京大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments Revision: The paper was accepted by Transactions of Machine Learning Research (TMLR)

2505.24523 2025-06-02 cs.CL cs.AI 73%

Stress-testing Machine Generated Text Detection: Shifting Language Models Writing Style to Fool Detectors

Andrea Pedrotti, Michele Papucci, Cristiano Ciaccio, Alessio Miaschi, Giovanni Puccetti, Felice Dell'Orletta, Andrea Esuli

机构 * Department of Computer Science, University of Pisa（比萨大学计算机科学系）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Accepted at Findings of ACL 2025

2505.20359 2025-05-30 cs.LG cs.AI 73%

Risk-aware Direct Preference Optimization under Nested Risk Measure

Lijun Zhang, Lin Li, Yajie Qi, Huizhong Song, Yaodong Yang, Jun Wang, Wei Wei

机构 * Key Laboratory of Computational Intelligence and Chinese Information Processing of Ministry of Education, School of Computer and Information Technology, Shanxi University（教育部计算智能与中文信息处理重点实验室，计算机与信息学院，山西大学）； Institute for AI, Peking University（人工智能研究院，北京大学）； University College London（伦敦大学学院）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2502.00814 2025-05-20 cs.LG cs.CL 73%

Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling

Jianfeng Cai, Jinhua Zhu, Ruopei Sun, Yue Wang, Li Li, Wengang Zhou, Houqiang Li

机构 * University of Science and Technology of China（中国科学技术大学）

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2410.12735 2025-04-29 cs.LG cs.CL 73%

CREAM: Consistency Regularized Self-Rewarding Language Models

Zhaoyang Wang, Weilei He, Zhiyuan Liang, Xuchao Zhang, Chetan Bansal, Ying Wei, Weitong Zhang, Huaxiu Yao

机构 * University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校）； Nanyang Technological University（南洋理工大学）； National University of Singapore（国立新加坡大学）； Microsoft Research（微软研究院）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments To appear at ICLR 2025

2504.14177 2025-04-22 cs.AI cs.CL cs.HC 73%

Direct Advantage Regression: Aligning LLMs with Online AI Reward

Li He, He Zhao, Stephen Wan, Dadong Wang, Lina Yao, Tongliang Liu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

2405.14953 2025-04-21 cs.LG cs.AI stat.ML 73%

MallowsPO: Fine-Tune Your LLM with Preference Dispersions

Haoxian Chen, Hanyang Zhao, Henry Lam, David Yao, Wenpin Tang

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2406.01660 2025-04-15 cs.LG cs.AI stat.ML 73%

Self-Improving Robust Preference Optimization

Eugene Choi, Arash Ahmadian, Matthieu Geist, Oilvier Pietquin, Mohammad Gheshlaghi Azar

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2410.10166 2025-04-03 cs.LG cs.AI 73%

Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models

Yongjin Yang, Sihyeon Kim, Hojung Jung, Sangmin Bae, SangMook Kim, Se-Young Yun, Kimin Lee

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments ICLR 2025; Project Page available at : https://sprain02.github.io/FiFA/

2503.24150 2025-04-01 cs.LG cs.AI cs.HC 73%

Learning a Canonical Basis of Human Preferences from Binary Ratings

Kailas Vodrahalli, Wei Wei, James Zou

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments 25 pages, 11 figures

2405.19316 2025-03-04 cs.LG cs.CL 73%

Robust Preference Optimization through Reward Model Distillation

Adam Fisch, Jacob Eisenstein, Vicky Zayats, Alekh Agarwal, Ahmad Beirami, Chirag Nagpal, Pete Shaw, Jonathan Berant

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2502.13922 2025-03-04 cs.CL cs.LG 73%

LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization

Guanzheng Chen, Xin Li, Michael Qizhe Shieh, Lidong Bing

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments ICLR 2025

1. 偏好对齐 3238 篇

Aligning Large Vision-Language Models by Deep Reinforcement Learning and Direct Preference Optimization

Igniting Creative Writing in Small Language Models: LLM-as-a-Judge versus Multi-Agent Refined Rewards

Mitigating Hallucinations in Multimodal LLMs via Object-aware Preference Optimization

WST: Weak-to-Strong Knowledge Transfer via Reinforcement Learning

Multimodal LLMs as Customized Reward Models for Text-to-Image Generation

SGPO: Self-Generated Preference Optimization based on Self-Improver

Reinforcement learning fine-tuning of language model for instruction following and math reasoning

FinDPO: Financial Sentiment Analysis for Algorithmic Trading through Preference Optimization of LLMs

IOPO: Empowering LLMs with Complex Instruction Following via Input-Output Preference Optimization

Tiny Reward Models

Learning From Crowdsourced Noisy Labels: A Signal Processing Perspective

Evaluating the Effectiveness of Direct Preference Optimization for Personalizing German Automatic Text Simplifications for Persons with Intellectual Disabilities

Symmetric Reinforcement Learning Loss for Robust Learning on Diverse Tasks and Model Scales

Trust Region Preference Approximation: A simple and stable reinforcement learning algorithm for LLM reasoning

CAMS: A CityGPT-Powered Agentic Framework for Urban Human Mobility Simulation

Intra-Trajectory Consistency for Reward Modeling

Rethinking Diverse Human Preference Learning through Principal Component Analysis

David and Goliath: Small One-step Model Beats Large Diffusion with Score Post-training

Diff-Instruct++: Training One-step Text-to-image Generator Model to Align with Human Preferences

Stress-testing Machine Generated Text Detection: Shifting Language Models Writing Style to Fool Detectors

Risk-aware Direct Preference Optimization under Nested Risk Measure

Disentangling Length Bias In Preference Learning Via Response-Conditioned Modeling

CREAM: Consistency Regularized Self-Rewarding Language Models

Direct Advantage Regression: Aligning LLMs with Online AI Reward

MallowsPO: Fine-Tune Your LLM with Preference Dispersions

Self-Improving Robust Preference Optimization

Automated Filtering of Human Feedback Data for Aligning Text-to-Image Diffusion Models

Learning a Canonical Basis of Human Preferences from Binary Ratings

Robust Preference Optimization through Reward Model Distillation

LongPO: Long Context Self-Evolution of Large Language Models through Short-to-Long Preference Optimization