大模型对齐与安全

MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimization

Yougang Lyu, Lingyong Yan, Zihan Wang, Dawei Yin, Pengjie Ren, Maarten de Rijke, Zhaochun Ren

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments ICLR 2025

Super(ficial)-alignment: Strong Models May Deceive Weak Models in Weak-to-Strong Generalization

Wenkai Yang, Shiqi Shen, Guangyao Shen, Wei Yao, Yong Liu, Zhi Gong, Yankai Lin, Ji-Rong Wen

专题命中 偏好对齐 ：alignment(title,abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICLR 2025, camera-ready version

Evaluating Defences against Unsafe Feedback in RLHF

Domenic Rosati, Giles Edkins, Harsh Raj, David Atanasov, Subhabrata Majumdar, Janarthanan Rajendran, Frank Rudzicz, Hassan Sajjad

专题命中 偏好对齐 ：RLHF(title,abstract)；safety(abstract)；分类 cs.CL、cs.LG

Learn Your Reference Model for Real Good Alignment

Alexey Gorbatovski, Boris Shaposhnikov, Alexey Malakhov, Nikita Surnachev, Yaroslav Aksenov, Ian Maksimov, Nikita Balagansky, Daniil Gavrilov

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Is Free Self-Alignment Possible?

Dyah Adila, Changho Shin, Yijing Zhang, Frederic Sala

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignment

Shengyang Sun, Yian Zhang, Alexander Bukharin, David Mosallanezhad, Jiaqi Zeng, Soumye Singhal, Gerald Shen, Adithya Renduchintala, Tugrul Konuk, Yi Dong, Zhilin Wang, Dmitry Chichkov, Olivier Delalleau, Oleksii Kuchaiev

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments 8 pages, 4 figures; update author names

Extracting and Understanding the Superficial Knowledge in Alignment

Runjin Chen, Gabriel Jacob Perin, Xuxi Chen, Xilun Chen, Yan Han, Nina S. T. Hirata, Junyuan Hong, Bhavya Kailkhura

专题命中 偏好对齐 ：alignment(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI

Jackpot! Alignment as a Maximal Lottery

Roberto-Rafael Maura-Rivero, Marc Lanctot, Francesco Visin, Kate Larson

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment

Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe

专题命中 偏好对齐 ：alignment(title)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments NAACL 2025

Style Outweighs Substance: Failure Modes of LLM Judges in Alignment Benchmarking

Benjamin Feuer, Micah Goldblum, Teresa Datta, Sanjana Nambiar, Raz Besaleli, Samuel Dooley, Max Cembalest, John P. Dickerson

专题命中 偏好对齐 ：alignment(title,abstract)；safety(abstract)；分类 cs.AI、cs.LG

Comments ICLR 2025

Inverse-RLignment: Large Language Model Alignment from Demonstrations through Inverse Reinforcement Learning

Hao Sun, Mihaela van der Schaar

专题命中 偏好对齐 ：alignment(title,abstract)；safety(abstract)；分类 cs.AI、cs.LG

Clone-Robust AI Alignment

Ariel D. Procaccia, Benjamin Schiffer, Shirley Zhang

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment

Prashant Trivedi, Souradip Chakraborty, Avinash Reddy, Vaneet Aggarwal, Amrit Singh Bedi, George K. Atia

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments 27 pages, Accepted in AAAI 2025

IIMedGPT: Promoting Large Language Model Capabilities of Medical Tasks by Efficient Human Preference Alignment

Yiming Zhang, Zheng Chang, Wentao Cai, MengXing Ren, Kang Yuan, Yining Sun, Zenghui Ding

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Northeastern Uni at Multilingual Counterspeech Generation: Enhancing Counter Speech Generation with LLM Alignment through Direct Preference Optimization

Sahil Wadhwa, Chengtian Xu, Haoming Chen, Aakash Mahalingam, Akankshya Kar, Divya Chaudhary

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments 10 pages, 6 tables, 1 figure, The First Workshop on Multilingual Counterspeech Generation (MCG) at The 31st International Conference on Computational Linguistics (COLING 2025)

Journal ref The First Workshop on Multilingual Counterspeech Generation (MCG) at The 31st International Conference on Computational Linguistics (COLING 2025)

Context-DPO: Aligning Language Models for Context-Faithfulness

Baolong Bi, Shaohan Huang, Yiwei Wang, Tianchi Yang, Zihan Zhang, Haizhen Huang, Lingrui Mei, Junfeng Fang, Zehao Li, Furu Wei, Weiwei Deng, Feng Sun, Qi Zhang, Shenghua Liu

专题命中 偏好对齐 ：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

PA-RAG: RAG Alignment via Multi-Perspective Preference Optimization

Jiayi Wu, Hengyi Cai, Lingyong Yan, Hao Sun, Xiang Li, Shuaiqiang Wang, Dawei Yin, Ming Gao

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignment

Zhuoran Jin, Hongbang Yuan, Tianyi Men, Pengfei Cao, Yubo Chen, Kang Liu, Jun Zhao

专题命中 偏好对齐 ：alignment(title,abstract)；trustworthy(abstract)；分类 cs.CL、cs.AI

Comments 26 pages, 12 figures, 6 tables

Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets

Duanyu Feng, Bowen Qin, Chen Huang, Youcheng Huang, Zheng Zhang, Wenqiang Lei

专题命中 偏好对齐 ：safety(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

Comments Our code is available at https://github.com/colfeng/Legend

Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hacking

Paria Rashidinejad, Yuandong Tian

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments 46 pages, 3 figures

Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer

Zhihan Liu, Miao Lu, Shenao Zhang, Boyi Liu, Hongyi Guo, Yingxiang Yang, Jose Blanchet, Zhaoran Wang

专题命中 偏好对齐 ：RLHF(title,abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments Accepted by The Thirty-Eighth Annual Conference on Neural Information Processing Systems. 31 pages, 7 figures

Noise Contrastive Alignment of Language Models with Explicit Rewards

Huayu Chen, Guande He, Lifan Yuan, Ganqu Cui, Hang Su, Jun Zhu

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2024

2D-DPO: Scaling Direct Preference Optimization with 2-Dimensional Supervision

Shilong Li, Yancheng He, Hui Huang, Xingyuan Bu, Jiaheng Liu, Hangyu Guo, Weixun Wang, Jihao Gu, Wenbo Su, Bo Zheng

专题命中 偏好对齐 ：DPO(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

Comments The first four authors contributed equally, 25 pages

PopAlign: Diversifying Contrasting Patterns for a More Comprehensive Alignment

Zekun Moore Wang, Shawn Wang, Kang Zhu, Jiaheng Liu, Ke Xu, Jie Fu, Wangchunshu Zhou, Wenhao Huang

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 28 pages

Anchored Alignment for Self-Explanations Enhancement

Luis Felipe Villa-Arenas, Ata Nizamoglu, Qianli Wang, Sebastian Möller, Vera Schmitt

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models

Yanjun Chen, Dawei Zhu, Yirong Sun, Xinghao Chen, Wei Zhang, Xiaoyu Shen

专题命中 偏好对齐 ：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

Comments 10 pages, 27 figures (including 18 in the appendix), submitted to EMNLP 2024

Progressively Label Enhancement for Large Language Model Alignment

Biao Liu, Ning Xu, Xin Geng

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

SYNFAC-EDIT: Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization

Prakamya Mishra, Zonghai Yao, Parth Vashisht, Feiyun Ouyang, Beining Wang, Vidhi Dhaval Mody, Hong Yu

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Equal contribution for the first two authors; To appear in proceedings of the Main Conference on Empirical Methods in Natural Language Processing (EMNLP) 2024

Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimization

Shitong Duan, Xiaoyuan Yi, Peng Zhang, Yan Liu, Zheng Liu, Tun Lu, Xing Xie, Ning Gu

专题命中 偏好对齐 ：alignment(title,abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Accepted by EMNLP 2024(Findings)

Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment

Yan Liu, Xiaoyuan Yi, Xiaokang Chen, Jing Yao, Jingwei Yi, Daoguang Zan, Zheng Liu, Xing Xie, Tsung-Yi Ho

专题命中 偏好对齐 ：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI