大模型对齐与安全

SPPD: Self-training with Process Preference Learning Using Dynamic Value Margin

Hao Yi, Qingyang Li, Yulan Hu, Fuzheng Zhang, Di Zhang, Yong Liu

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.AI

Balancing the Budget: Understanding Trade-offs Between Supervised and Preference-Based Finetuning

Mohit Raghavendra, Junmo Kang, Alan Ritter

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.LG

Preference Optimization for Reasoning with Pseudo Feedback

Fangkai Jiao, Geyang Guo, Xingxing Zhang, Nancy F. Chen, Shafiq Joty, Furu Wei

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments 28 pages, 11 figures. ICLR 2025

On-Board Vision-Language Models for Personalized Autonomous Vehicle Motion Control: System Design and Real-World Validation

Can Cui, Zichong Yang, Yupeng Zhou, Juntong Peng, Sung-Yeon Park, Cong Zhang, Yunsheng Ma, Xu Cao, Wenqian Ye, Yiheng Feng, Jitesh Panchal, Lingxi Li, Yaobin Chen, Ziran Wang

专题命中 偏好对齐 ：safety(abstract)；分类 cs.AI

Comments arXiv admin note: substantial text overlap with arXiv:2410.15281

EmoAssist: Emotional Assistant for Visual Impairment Community

Xingyu Qi, He Li, Linjie Li, Zhenyu Wu

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CY

Self-Consistency of the Internal Reward Models Improves Self-Rewarding Language Models

Xin Zhou, Yiwen Guo, Ruotian Ma, Tao Gui, Qi Zhang, Xuanjing Huang

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI

Optimizing Knowledge Integration in Retrieval-Augmented Generation with Self-Selection

Yan Weng, Fengbin Zhu, Tong Ye, Haoyan Liu, Fuli Feng, Tat-Seng Chua

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments 12 pages, 6 figures

Preference Optimization via Contrastive Divergence: Your Reward Model is Secretly an NLL Estimator

Zhuotong Chen, Fang Liu, Xuan Zhu, Yanjun Qi, Mohammad Ghavamzadeh

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI

ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization

Yinjie Wang, Ling Yang, Guohao Li, Mengdi Wang, Bryon Aragam

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments Project: https://github.com/Gen-Verse/ScoreFlow

PILAF: Optimal Human Preference Sampling for Reward Modeling

Yunzhen Feng, Ariel Kwiatkowski, Kunhao Zheng, Julia Kempe, Yaqi Duan

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.LG

Preference VLM: Leveraging VLMs for Scalable Preference-Based Reinforcement Learning

Udita Ghosh, Dripta S. Raychaudhuri, Jiachen Li, Konstantinos Karydis, Amit Roy-Chowdhury

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.LG

RoVRM: A Robust Visual Reward Model Optimized via Auxiliary Textual Preference Data

Chenglong Wang, Yang Gan, Yifu Huo, Yongyu Mu, Murun Yang, Qiaozhi He, Tong Xiao, Chunliang Zhang, Tongran Liu, Quan Du, Di Yang, Jingbo Zhu

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Comments Accepted by AAAI 2025

CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

Jinlan Fu, Shenzhen Huangfu, Hao Fei, Xiaoyu Shen, Bryan Hooi, Xipeng Qiu, See-Kiong Ng

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments Accepted by ICLR 2025

Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives

Hao Sun, Yunyi Shen, Jean-Francois Ton

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI

Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis

Qining Zhang, Honghao Wei, Lei Ying

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.LG

DNA 1.0 Technical Report

Jungyup Lee, Jemin Kim, Sang Park, SeungJae Lee

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion

Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Gheshlaghi Azar, Olivier Pietquin, Matthieu Geist

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.LG

Comments EMNLP 2024

SYNAPSE: SYmbolic Neural-Aided Preference Synthesis Engine

Sadanand Modak, Noah Patton, Isil Dillig, Joydeep Biswas

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.LG

Comments Accepted (oral) at AAAI 25

MM-PhyRLHF: Reinforcement Learning Framework for Multimodal Physics Question-Answering

Janak Kapuriya, Chhavi Kirtani, Apoorv Singh, Jay Saraf, Naman Lal, Jatin Kumar, Adarsh Raj Shivam, Astha Verma, Avinash Anand, Rajiv Ratn Shah

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.AI

LoRA-LiteE: A Computationally Efficient Framework for Chatbot Preference-Tuning

Yahe Yang, Chunliang Tao, Xiaojing Fan

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.CL

Distilling Desired Comments for Enhanced Code Review with Large Language Models

Yongda Yu, Lei Zhang, Guoping Rong, Haifeng Shen, Jiahao Zhang, Haoxiang Yan, Guohao Shi, Dong Shao, Ruiqi Pan, Yuan Li, Qiushi Wang, Zhao Tian

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.AI

Comments 12 pages, 9 figures

Qwen2.5 Technical Report

Qwen, :, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Mei Li, Mingfeng Xue, Pei Zhang, Qin Zhu, Rui Men, Runji Lin, Tianhao Li, Tianyi Tang, Tingyu Xia, Xingzhang Ren, Xuancheng Ren, Yang Fan, Yang Su, Yichang Zhang, Yu Wan, Yuqiong Liu, Zeyu Cui, Zhenru Zhang, Zihan Qiu

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Plug-and-Play Training Framework for Preference Optimization

Jingyuan Ma, Rui Li, Zheng Li, Lei Sha, Zhifang Sui

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments 12 pages, 9 figures

Online Learning from Strategic Human Feedback in LLM Fine-Tuning

Shugang Hao, Lingjie Duan

专题命中 偏好对齐 ：RLHF(abstract)；分类 cs.AI

Teaching LLMs to Refine with Tools

Dian Yu, Yuheng Zhang, Jiahao Xu, Tian Liang, Linfeng Song, Zhaopeng Tu, Haitao Mi, Dong Yu

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

The Superalignment of Superhuman Intelligence with Large Language Models

Minlie Huang, Yingkang Wang, Shiyao Cui, Pei Ke, Jie Tang

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Comments Under review of Science China

WEPO: Web Element Preference Optimization for LLM-based Web Navigation

Jiarun Liu, Jia Hao, Chunhong Zhang, Zheng Hu

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.CL

Comments Published at AAAI 2025

Learning to Solve Domain-Specific Calculation Problems with Knowledge-Intensive Programs Generator

Chengyuan Liu, Shihang Wang, Lizhi Qing, Jun Lin, Ji Zhang, Fei Wu, Kun Kuang

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Comments Under review

Learning to Reason via Self-Iterative Process Feedback for Small Language Models

Kaiyuan Chen, Jin Wang, Xuejie Zhang

专题命中 偏好对齐 ：alignment(abstract)；分类 cs.CL

Comments Accepted by COLING 2025

DSTC: Direct Preference Learning with Only Self-Generated Tests and Code to Improve Code LMs

Zhihan Liu, Shenao Zhang, Yongfei Liu, Boyi Liu, Yingxiang Yang, Zhaoran Wang

专题命中 偏好对齐 ：DPO(abstract)；分类 cs.AI