大模型对齐与安全 - arXivDaily 专题

2505.10597 2025-05-20 cs.LG cs.AI cs.CL 82%

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Tao Gui, Qi Zhang

机构 * NLP Group, Fudan University（复旦大学自然语言处理组）； Hong Kong Baptist University（香港 Baptist 大学）； LLM Team, Meituan Inc（美团公司大语言模型团队）

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2505.07271 2025-05-13 cs.CL cs.AI cs.LG 82%

On the Robustness of Reward Models for Language Model Alignment

Jiwoo Hong, Noah Lee, Eunki Kim, Guijin Son, Woojin Chung, Aman Gupta, Shao Tang, James Thorne

专题命中偏好对齐：alignment(title)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICML 2025

2406.14088 2025-04-25 cs.DC cs.AI cs.CL cs.LG 82%

ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation

Zhiyu Mei, Wei Fu, Kaiwei Li, Guangju Wang, Huanchen Zhang, Yi Wu

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 11 pages (20 pages with references and the appendix), 17 figures. Accepted by MLSys 25

2504.10329 2025-04-22 cs.CV 82%

InstructEngine: Instruction-driven Text-to-Image Alignment

Xingyu Lu, Yuhang Hu, YiFan Zhang, Kaiyu Jiang, Changyi Liu, Tianke Zhang, Jinpeng Wang, Chun Yuan, Bin Wen, Fan Yang, Tingting Gao, Di Zhang

机构 * Tsinghua University（清华大学）； Kuaishou Technology（快手科技）； CASIA（中国科学院自动化研究所）

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)

Comments 8 pages, 7 figures

2407.07880 2025-04-21 cs.LG cs.AI cs.CL 82%

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, Xiangnan He

专题命中偏好对齐：alignment(title)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2503.16480 2025-03-24 cs.HC cs.AI cs.CL cs.CY 82%

Human Preferences for Constructive Interactions in Language Model Alignment

Yara Kyrychenko, Jon Roozenbeek, Brandon Davidson, Sander van der Linden, Ramit Debnath

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.CY

Comments 1 Figure, 1 Table, 11 pages

2503.01864 2025-03-05 cs.LG cs.AI cs.CL 82%

Larger or Smaller Reward Margins to Select Preferences for Alignment?

Kexin Huang, Junkang Wu, Ziqian Chen, Xue Wang, Jinyang Gao, Bolin Ding, Jiancan Wu, Xiangnan He, Xiang Wang

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2407.06057 2025-03-05 cs.CL cs.AI cs.LG 82%

Variational Best-of-N Alignment

Afra Amini, Tim Vieira, Elliott Ash, Ryan Cotterell

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at ICLR 2025

2406.04412 2025-03-05 cs.LG cs.AI cs.CL 82%

Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

Dongyoung Kim, Kimin Lee, Jinwoo Shin, Jaehyung Kim

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2025 Oral Presentation, 22 pages

2501.09695 2025-03-04 cs.CV 82%

Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key

Zhihe Yang, Xufang Luo, Dongqi Han, Yunjian Xu, Dongsheng Li

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)

Comments Accepted by CVPR 2025

2502.18411 2025-03-04 cs.CV 82%

OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference

Xiangyu Zhao, Shengyuan Ding, Zicheng Zhang, Haian Huang, Maosong Cao, Weiyun Wang, Jiaqi Wang, Xinyu Fang, Wenhai Wang, Guangtao Zhai, Haodong Duan, Hua Yang, Kai Chen

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

2502.20356 2025-02-28 cs.CL cs.AI cs.LG 82%

Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs

Kuan Lok Zhou, Jiayi Chen, Siddharth Suresh, Reuben Narad, Timothy T. Rogers, Lalit K Jain, Robert D Nowak, Bob Mankoff, Jifan Zhang

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2501.00560 2025-02-12 cs.CL cs.AI cs.LG 82%

Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference

Mingqi Gao, Yixin Liu, Xinyu Hu, Xiaojun Wan, Jonathan Bragg, Arman Cohan

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Findings of NAACL 2025

2405.15973 2025-02-11 cs.CV cs.AI cs.CL cs.LG 82%

Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement

Xiyao Wang, Jiuhai Chen, Zhaoyang Wang, Yuhang Zhou, Yiyang Zhou, Huaxiu Yao, Tianyi Zhou, Tom Goldstein, Parminder Bhatia, Furong Huang, Cao Xiao

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NAACL 2025 Findings

2403.09472 2024-12-11 cs.LG cs.AI cs.CL 82%

Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

Zhiqing Sun, Longhui Yu, Yikang Shen, Weiyang Liu, Yiming Yang, Sean Welleck, Chuang Gan

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at NeurIPS 2024

2405.07863 2024-11-13 cs.LG cs.AI cs.CL stat.ML 82%

RLHF Workflow: From Reward Modeling to Online RLHF

Hanze Dong, Wei Xiong, Bo Pang, Haoxiang Wang, Han Zhao, Yingbo Zhou, Nan Jiang, Doyen Sahoo, Caiming Xiong, Tong Zhang

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published in Transactions on Machine Learning Research (09/2024)

2409.09774 2024-11-07 cs.CV 82%

Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through $f$-divergence Minimization

Haoyuan Sun, Bo Xia, Yongzhe Chang, Xueqian Wang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

Comments 34 pages

2410.14872 2024-10-24 cs.LG cs.AI cs.CL 82%

How to Evaluate Reward Models for RLHF

Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2406.12168 2024-10-23 cs.LG cs.AI cs.CL 82%

BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment

Wenda Xu, Jiachen Li, William Yang Wang, Lei Li

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Wenda Xu and Jiachen Li contributed equally. Accepted by EMNLP 2024

2402.10207 2024-10-17 cs.LG cs.AI cs.CL 82%

Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment

Rui Yang, Xiaoman Pan, Feng Luo, Shuang Qiu, Han Zhong, Dong Yu, Jianshu Chen

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by ICML 2024

2310.11523 2024-10-16 cs.LG cs.AI cs.CL 82%

Group Preference Optimization: Few-Shot Alignment of Large Language Models

Siyan Zhao, John Dang, Aditya Grover

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments accepted at ICLR 2024, code at https://github.com/jamqd/Group-Preference-Optimization

2312.11819 2024-10-15 cs.LG cs.AI cs.CL 82%

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training

Youshao Xiao, Zhenglei Zhou, Fagui Mao, Weichang Wu, Shangchun Zhao, Lin Ju, Lei Liang, Xiaolu Zhang, Jun Zhou

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2410.07513 2024-10-11 cs.LG cs.AI cs.CL 82%

Evolutionary Contrastive Distillation for Language Model Alignment

Julian Katz-Samuels, Zheng Li, Hyokun Yun, Priyanka Nigam, Yi Xu, Vaclav Petricek, Bing Yin, Trishul Chilimbi

专题命中偏好对齐：alignment(title)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.03469 2024-10-08 cs.LG cs.AI cs.CL 82%

Rethinking the Role of Proxy Rewards in Language Model Alignment

Sungdong Kim, Minjoon Seo

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to EMNLP 2024 main conference

2410.03212 2024-10-07 cs.IR 82%

Data-Efficient Massive Tool Retrieval: A Reinforcement Learning Approach for Query-Tool Alignment with Language Models

Yuxiang Zhang, Xin Fan, Junjie Wang, Chongxian Chen, Fan Mo, Tetsuya Sakai, Hayato Yamana

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)

2409.10164 2024-09-17 cs.LG cs.AI cs.CL 82%

Quantile Regression for Distributional Reward Models in RLHF

Nicolai Dorka

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2408.06266 2024-09-17 cs.LG cs.AI cs.CL 82%

Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment

Karel D'Oosterlinck, Winnie Xu, Chris Develder, Thomas Demeester, Amanpreet Singh, Christopher Potts, Douwe Kiela, Shikib Mehri

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

2309.00267 2024-09-04 cs.CL cs.AI cs.LG 82%

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

Harrison Lee, Samrat Phatale, Hassan Mansoor, Thomas Mesnard, Johan Ferret, Kellie Lu, Colton Bishop, Ethan Hall, Victor Carbune, Abhinav Rastogi, Sushant Prakash

专题命中偏好对齐：RLHF(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Presented at ICML 2024

Journal ref Proceedings of the 41st International Conference on Machine Learning, PMLR 235:26874-26901, 2024

2408.12579 2024-08-23 cs.CL cs.AI cs.HC cs.IR cs.LG 82%

RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment

Xiaohan Wang, Xiaoyan Yang, Yuqi Zhu, Yue Shen, Jian Wang, Peng Wei, Lei Liang, Jinjie Gu, Huajun Chen, Ningyu Zhang

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Ongoing work

2404.10501 2024-08-22 cs.CV cs.AI cs.CL cs.LG 82%

Self-Supervised Visual Preference Alignment

Ke Zhu, Zheng Ge, Liang Zhao, Xiangyu Zhang

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments MM2024 oral

1. 偏好对齐 3229 篇

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

On the Robustness of Reward Models for Language Model Alignment

ReaL: Efficient RLHF Training of Large Language Models with Parameter Reallocation

InstructEngine: Instruction-driven Text-to-Image Alignment

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

Human Preferences for Constructive Interactions in Language Model Alignment

Larger or Smaller Reward Margins to Select Preferences for Alignment?

Variational Best-of-N Alignment

Spread Preference Annotation: Direct Preference Judgment for Efficient LLM Alignment

Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key

OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference

Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs

Re-evaluating Automatic LLM System Ranking for Alignment with Human Preference

Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement

Easy-to-Hard Generalization: Scalable Alignment Beyond Human Supervision

RLHF Workflow: From Reward Modeling to Online RLHF

Generalizing Alignment Paradigm of Text-to-Image Generation with Preferences through $f$-divergence Minimization

How to Evaluate Reward Models for RLHF

BPO: Staying Close to the Behavior LLM Creates Better Online LLM Alignment

Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustment

Group Preference Optimization: Few-Shot Alignment of Large Language Models

An Adaptive Placement and Parallelism Framework for Accelerating RLHF Training

Evolutionary Contrastive Distillation for Language Model Alignment

Rethinking the Role of Proxy Rewards in Language Model Alignment

Data-Efficient Massive Tool Retrieval: A Reinforcement Learning Approach for Query-Tool Alignment with Language Models

Quantile Regression for Distributional Reward Models in RLHF

Anchored Preference Optimization and Contrastive Revisions: Addressing Underspecification in Alignment

RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback

RuleAlign: Making Large Language Models Better Physicians with Diagnostic Rule Alignment

Self-Supervised Visual Preference Alignment