大模型对齐与安全 - arXivDaily 专题

2506.12446 2025-07-01 cs.CL 79%

From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment

Bin Xie, Bingbing Xu, Yige Yuan, Shengmao Zhu, Huawei Shen

机构 * State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（人工智能安全国家重点实验室、计算技术研究所、中国科学院）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

2410.03145 2025-07-01 cs.CL 79%

Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback

Kyuyoung Kim, Ah Jeong Seo, Hao Liu, Jinwoo Shin, Kimin Lee

机构 * KAIST（韩国科学技术院）； UC Berkeley（加州大学伯克利分校）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments EMNLP 2024 Findings

2506.18245 2025-06-24 cs.CR cs.AI cs.SE 79%

Smart-LLaMA-DPO: Reinforced Large Language Model for Explainable Smart Contract Vulnerability Detection

Lei Yu, Zhirong Huang, Hang Yuan, Shiqi Cheng, Li Yang, Fengjun Zhang, Chenjie Shen, Jiajia Ma, Jingyuan Zhang, Junyi Lu, Chun Zuo

机构 * Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所）； Sinosoft Company Limited（软通公司）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.AI

Comments Accepted to ISSTA 2025

2506.17951 2025-06-24 cs.CL 79%

A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignment

Quanwei Tang, Sophia Yat Mei Lee, Junshuang Wu, Dong Zhang, Shoushan Li, Erik Cambria, Guodong Zhou

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments acl 2025 findings

2506.17834 2025-06-24 cs.AI cs.HC 79%

Reflective Verbal Reward Design for Pluralistic Alignment

Carter Blair, Kate Larson, Edith Law

机构 * University of Waterloo（滑铁卢大学）

专题命中偏好对齐：alignment(title)；RLHF(abstract)；分类 cs.AI

Comments 9 pages, 3 figures, accepted to the IJCAI 2025 Human-Centred AI track. Project repository at: https://osf.io/8yxf2/

2502.07599 2025-06-09 cs.CL 79%

DPO-Shift: Shifting the Distribution of Direct Preference Optimization

Xiliang Yang, Feng Jiang, Qianen Zhang, Lei Zhao, Xiao Li

机构 * David S. Hippocampus Department of Computer Science（计算机科学系）

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL

2506.02460 2025-06-04 cs.CL 79%

MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework

Yupeng Qi, Ziyu Lyu, Min Yang, Yanlin Wang, Lu Bai, Lixin Cui

机构 * Sun Yat-sen University（中山大学）； Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（深圳先进技术研究院、中国科学院）； Beijing Normal University（北京师范大学）； Central University of Finance and Economics（中央财经大学）

专题命中偏好对齐：safety(title,abstract)；分类 cs.CL

2501.19358 2025-06-03 cs.LG 79%

The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

Yuchun Miao, Sen Zhang, Liang Ding, Yuqi Zhang, Lefei Zhang, Dacheng Tao

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.LG

Comments The paper has been accepted by ICML 2025

2505.23927 2025-06-02 cs.LG 79%

Thompson Sampling in Online RLHF with General Function Approximation

Songtao Feng, Jie Fu

机构 * Department of Electrical and Computer Engineering, University of Florida（电气与计算机工程系，佛罗里达大学）

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.LG

2505.21395 2025-05-28 cs.LG 79%

Square$χ$PO: Differentially Private and Robust $χ^2$-Preference Optimization in Offline Direct Alignment

Xingyu Zhou, Yulian Wu, Wenqian Weng, Francesco Orabona

专题命中偏好对齐：alignment(title,abstract)；分类 cs.LG

2505.20871 2025-05-28 cs.CL 79%

Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG

Xin Sun, Jianan Xie, Zhongqi Chen, Qiang Liu, Shu Wu, Yuehe Chen, Bowen Song, Weiqiang Wang, Zilei Wang, Liang Wang

专题命中偏好对齐：alignment(title)；DPO(abstract)；分类 cs.CL

Comments ACL 2025 main

2505.20556 2025-05-28 cs.LG 79%

Learning a Pessimistic Reward Model in RLHF

Yinglun Xu, Hangoo Kang, Tarun Suresh, Yuxuan Wan, Gagandeep Singh

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.LG

2505.00038 2025-05-21 cs.CL 79%

HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation

Cristina Garbacea, Chenhao Tan

机构 * University of Chicago（芝加哥大学）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

2505.12845 2025-05-20 cs.AI 79%

Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks

Ruopei Sun, Jianfeng Cai, Jinhua Zhu, Kangwen Zhao, Dongyun Xue, Wengang Zhou, Li Li, Houqiang Li

机构 * University of Science and Technology of China（中国科学技术大学）

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.AI

2406.09574 2025-05-19 cs.LG 79%

Online Bandit Learning with Offline Preference Data for Improved RLHF

Akhil Agnihotri, Rahul Jain, Deepak Ramachandran, Zheng Wen

机构 * University of Southern California（南加州大学）； Google DeepMind（谷歌DeepMind）； USC（南加州大学）

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.LG

2505.03059 2025-05-07 cs.CL 79%

Improving Model Alignment Through Collective Intelligence of Open-Source LLMS

Junlin Wang, Roy Xie, Shang Zhu, Jue Wang, Ben Athiwaratkun, Bhuwan Dhingra, Shuaiwen Leon Song, Ce Zhang, James Zou

机构 * Duke University（杜克大学）； Together AI ； University of Chicago（芝加哥大学）； Stanford University（斯坦福大学）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments ICML 2025

2503.17003 2025-05-06 cs.CL 79%

A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications

Jian Guan, Junfei Wu, Jia-Nan Li, Chuanqi Cheng, Wei Wu

机构 * Ant Group（蚂蚁集团）； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所）； Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments Survey paper; 11 pages; Literature reviewed up to ICLR 2025

2410.01957 2025-05-02 cs.CL 79%

Challenges and Future Directions of Data-Centric AI Alignment

Min-Hsuan Yeh, Jeffrey Wang, Xuefeng Du, Seongheon Park, Leitian Tao, Shawn Im, Yixuan Li

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments ICML 2025

2409.08813 2025-04-28 cs.CL 79%

Your Weak LLM is Secretly a Strong Teacher for Alignment

Leitian Tao, Yixuan Li

机构 * Department of Computer Sciences, University of Wisconsin-Madison（计算机科学系，威斯康星大学麦迪逊分校）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments Accepted by ICLR 2025

2504.15719 2025-04-23 cs.AI 79%

Implementing Rational Choice Functions with LLMs and Measuring their Alignment with User Preferences

Anna Karnysheva, Christian Drescher, Dietrich Klakow

机构 * Spoken Language Systems (LSV)（语音语言系统（LSV））； Saarland University（萨尔兰大学）； Mercedes-Benz AG（梅赛德斯-奔驰集团）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.AI

2410.14148 2025-04-22 cs.CV cs.CL 79%

Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment

Chenhang Cui, An Zhang, Yiyang Zhou, Zhaorun Chen, Gelei Deng, Huaxiu Yao, Tat-Seng Chua

机构 * National University of Singapore（新加坡国立大学）； UNC-Chapel Hill（北卡罗来纳大学教堂山分校）； University of Chicago（芝加哥大学）； Nanyang Technological University（南洋理工大学）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments 23 pages; Published as a conference paper at ICLR 2025

2504.07070 2025-04-10 cs.CL 79%

A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models

Zhouhang Xie, Junda Wu, Yiran Shen, Yu Xia, Xintong Li, Aaron Chang, Ryan Rossi, Sachin Kumar, Bodhisattwa Prasad Majumder, Jingbo Shang, Prithviraj Ammanabrolu, Julian McAuley

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

2504.04657 2025-04-08 cs.LG 79%

ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback

Tasnia Rahman, Sathish A. P. Kumar, Sumit Jha, Arvind Ramanathan

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.LG

Comments 9 pages, 3 figures

2410.09893 2025-04-07 cs.CL 79%

RMB: Comprehensively Benchmarking Reward Models in LLM Alignment

Enyu Zhou, Guodong Zheng, Binghai Wang, Zhiheng Xi, Shihan Dou, Rong Bao, Wei Shen, Limao Xiong, Jessica Fan, Yurong Mou, Rui Zheng, Tao Gui, Qi Zhang, Xuanjing Huang

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments Accepted by ICLR2025

2503.20105 2025-03-27 cs.AI cs.RO 79%

Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrations

Ran Tian, Kratarth Goel

专题命中偏好对齐：alignment(title,abstract)；分类 cs.AI

Comments ICLR 2025 Spotlight

2503.16094 2025-03-21 cs.CL 79%

Cultural Alignment in Large Language Models Using Soft Prompt Tuning

Reem I. Masoud, Martin Ferianc, Philip Treleaven, Miguel Rodrigues

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

2503.08035 2025-03-12 cs.CL 79%

Group Preference Alignment: Customized LLM Response Generation from In-Situ Conversations

Ishani Mondal, Jack W. Stokes, Sujay Kumar Jauhar, Longqi Yang, Mengting Wan, Xiaofeng Xu, Xia Song, Jennifer Neville

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments 23 pages

2503.04858 2025-03-10 cs.CV cs.AI 79%

SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winner

Kejia Chen, Jiawen Zhang, Jiacong Hu, Jiazhen Yang, Jian Lou, Zunlei Feng, Mingli Song

专题命中偏好对齐：alignment(title,abstract)；分类 cs.AI

2502.17927 2025-03-06 cs.CL 79%

Advantage-Guided Distillation for Preference Alignment in Small Language Models

Shiping Gao, Fanqi Wan, Jiajian Guo, Xiaojun Quan, Qifan Wang

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL

Comments Accepted by ICLR 2025(spotlight)

2410.11817 2025-03-04 cs.CV cs.LG cs.MM 79%

Improving Long-Text Alignment for Text-to-Image Diffusion Models

Luping Liu, Chao Du, Tianyu Pang, Zehan Wang, Chongxuan Li, Dong Xu

专题命中偏好对齐：alignment(title,abstract)；分类 cs.LG

Journal ref International Conference on Learning Representations (ICLR 2025)

1. 偏好对齐 3235 篇

From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignment

Margin Matching Preference Optimization: Enhanced Model Alignment with Granular Feedback

Smart-LLaMA-DPO: Reinforced Large Language Model for Explainable Smart Contract Vulnerability Detection

A Comprehensive Graph Framework for Question Answering with Mode-Seeking Preference Alignment

Reflective Verbal Reward Design for Pluralistic Alignment

DPO-Shift: Shifting the Distribution of Direct Preference Optimization

MidPO: Dual Preference Optimization for Safety and Helpfulness in Large Language Models via a Mixture of Experts Framework

The Energy Loss Phenomenon in RLHF: A New Perspective on Mitigating Reward Hacking

Thompson Sampling in Online RLHF with General Function Approximation

Square$χ$PO: Differentially Private and Robust $χ^2$-Preference Optimization in Offline Direct Alignment

Divide-Then-Align: Honest Alignment based on the Knowledge Boundary of RAG

Learning a Pessimistic Reward Model in RLHF

HyPerAlign: Interpretable Personalized LLM Alignment via Hypothesis Generation

Multi-Level Aware Preference Learning: Enhancing RLHF for Complex Multi-Instruction Tasks

Online Bandit Learning with Offline Preference Data for Improved RLHF

Improving Model Alignment Through Collective Intelligence of Open-Source LLMS

A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applications

Challenges and Future Directions of Data-Centric AI Alignment

Your Weak LLM is Secretly a Strong Teacher for Alignment

Implementing Rational Choice Functions with LLMs and Measuring their Alignment with User Preferences

Fine-Grained Verifiers: Preference Modeling as Next-token Prediction in Vision-Language Alignment

A Survey on Personalized and Pluralistic Preference Alignment in Large Language Models

ACE-RLHF: Automated Code Evaluation and Socratic Feedback Generation Tool using Large Language Models and Reinforcement Learning with Human Feedback

RMB: Comprehensively Benchmarking Reward Models in LLM Alignment

Direct Post-Training Preference Alignment for Multi-Agent Motion Generation Models Using Implicit Feedback from Pre-training Demonstrations

Cultural Alignment in Large Language Models Using Soft Prompt Tuning

Group Preference Alignment: Customized LLM Response Generation from In-Situ Conversations

SHAPE : Self-Improved Visual Preference Alignment by Iteratively Generating Holistic Winner

Advantage-Guided Distillation for Preference Alignment in Small Language Models

Improving Long-Text Alignment for Text-to-Image Diffusion Models