大模型对齐与安全

$β$-DPO: Direct Preference Optimization with Dynamic $β$

Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jinyang Gao, Bolin Ding, Xiang Wang, Xiangnan He

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.AI、cs.LG

Comments NeurIPS 2024

Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing

Zhangchen Xu, Fengqing Jiang, Luyao Niu, Yuntian Deng, Radha Poovendran, Yejin Choi, Bill Yuchen Lin

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Link: https://magpie-align.github.io/

Investigating on RLHF methodology

Alexey Kutalev, Sergei Markoff

专题命中 偏好对齐 ：RLHF(title)；alignment(abstract)；分类 cs.AI、cs.LG

Comments 23 pages, 6 figures, 6 tables

Systematic Characterization of the Effectiveness of Alignment in Large Language Models for Categorical Decisions

Isaac Kohane

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments 19 pages (without Appendix) Appendix 7 pages. 7 Figures

Value Alignment from Unstructured Text

Inkit Padhi, Karthikeyan Natesan Ramamurthy, Prasanna Sattigeri, Manish Nagireddy, Pierre Dognin, Kush R. Varshney

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.LG

Strong and weak alignment of large language models with human values

Mehdi Khamassi, Marceau Nahon, Raja Chatila

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted for publication in Scientific Reports, special issue on AI aligment

Understanding the Learning Dynamics of Alignment with Human Feedback

Shawn Im, Yixuan Li

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

A Long Way to Go: Investigating Length Correlations in RLHF

Prasann Singhal, Tanya Goyal, Jiacheng Xu, Greg Durrett

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.LG

Comments 21 pages, 13 figures, Accepted to COLM 2024

Prototypical Reward Network for Data-Efficient RLHF

Jinghan Zhang, Xiting Wang, Yiqiao Jin, Changyu Chen, Xinhao Zhang, Kunpeng Liu

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2024

LANE: Logic Alignment of Non-tuning Large Language Models and Online Recommendation Systems for Explainable Reason Generation

Hongke Zhao, Songming Zheng, Likang Wu, Bowen Yu, Jing Wang

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.LG

Knowledgeable Preference Alignment for LLMs in Domain-specific Question Answering

Yichi Zhang, Zhuo Chen, Yin Fang, Yanxi Lu, Fangming Li, Wen Zhang, Huajun Chen

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2024 (Findings). Code is available at https://github.com/zjukg/KnowPAT

Group Robust Preference Optimization in Reward-free RLHF

Shyam Sundhar Ramesh, Yifan Hu, Iason Chaimalas, Viraj Mehta, Pier Giuseppe Sessa, Haitham Bou Ammar, Ilija Bogunovic

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.LG

Comments Preprint

Offline Regularised Reinforcement Learning for Large Language Models Alignment

Pierre Harvey Richemond, Yunhao Tang, Daniel Guo, Daniele Calandriello, Mohammad Gheshlaghi Azar, Rafael Rafailov, Bernardo Avila Pires, Eugene Tarassov, Lucas Spangher, Will Ellsworth, Aliaksei Severyn, Jonathan Mallinson, Lior Shani, Gil Shamir, Rishabh Joshi, Tianqi Liu, Remi Munos, Bilal Piot

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

AI Alignment with Changing and Influenceable Reward Functions

Micah Carroll, Davis Foote, Anand Siththaranjan, Stuart Russell, Anca Dragan

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Comments Accepted to ICML 2024

RLHF from Heterogeneous Feedback via Personalization and Preference Aggregation

Chanwoo Park, Mingyang Liu, Dingwen Kong, Kaiqing Zhang, Asuman Ozdaglar

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Comments Added experiments

Adversarial DPO: Harnessing Harmful Data for Reducing Toxicity with Minimal Impact on Coherence and Evasiveness in Dialogue Agents

San Kim, Gary Geunbae Lee

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 7 figures, accepted to NAACL findings 2024

Provably Robust DPO: Aligning Language Models with Noisy Feedback

Sayak Ray Chowdhury, Anush Kini, Nagarajan Natarajan

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.CL、cs.LG

Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, Wenqiang Lei

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.CL、cs.AI

Comments Draft version

Removing RLHF Protections in GPT-4 via Fine-Tuning

Qiusi Zhan, Richard Fang, Rohan Bindu, Akul Gupta, Tatsunori Hashimoto, Daniel Kang

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to NAACL 2024. (7 pages)

Scaling Data Diversity for Fine-Tuning Language Models in Human Alignment

Feifan Song, Bowen Yu, Hao Lang, Haiyang Yu, Fei Huang, Houfeng Wang, Yongbin Li

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by LREC-COLING 2024

Improving Generalization of Alignment with Human Preferences through Group Invariant Learning

Rui Zheng, Wei Shen, Yuan Hua, Wenbin Lai, Shihan Dou, Yuhao Zhou, Zhiheng Xi, Xiao Wang, Haoran Huang, Tao Gui, Qi Zhang, Xuanjing Huang

专题命中 偏好对齐 ：alignment(title)；RLHF(abstract)；分类 cs.AI、cs.LG

Synthetic Imitation Edit Feedback for Factual Alignment in Clinical Summarization

Prakamya Mishra, Zonghai Yao, Shuwei Chen, Beining Wang, Rohan Mittal, Hong Yu

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Comments To appear in NeurIPS 2023 Workshop SyntheticData4ML

Constructive Large Language Models Alignment with Diverse Feedback

Tianshu Yu, Ting-En Lin, Yuchuan Wu, Min Yang, Fei Huang, Yongbin Li

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.CL、cs.AI

Reinforcement Learning in the Era of LLMs: What is Essential? What is needed? An RL Perspective on RLHF, Prompting, and Beyond

Hao Sun

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.AI、cs.LG

Dynamic value alignment through preference aggregation of multiple objectives

Marcin Korecki, Damian Dailisan, Cesare Carissimo

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Stabilizing RLHF through Advantage Model and Selective Rehearsal

Baolin Peng, Linfeng Song, Ye Tian, Lifeng Jin, Haitao Mi, Dong Yu

专题命中 偏好对齐 ：RLHF(title,abstract)；分类 cs.CL、cs.AI

Comments 9 pages, working in progress

Scalable agent alignment via reward modeling: a research direction

Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini, Shane Legg

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.AI、cs.LG

Alignment of Diffusion Models: Fundamentals, Challenges, and Future

扩散模型对齐：基础、挑战与未来

Buhua Liu, Shitong Shao, Bao Li, Lichen Bai, Zhiqiang Xu, Haoyi Xiong, James Kwok, Sumi Helal, Zeke Xie

机构 * The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）） ； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） ； Mohamed bin Zayed University of Artificial Intelligence（穆罕默德·本·扎耶德人工智能大学） ； Baidu Inc.（百度公司） ； The University of Bologna（博洛尼亚大学） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.LG

AI总结 本文综述了扩散模型对齐的基础、挑战及未来方向，探讨了对齐技术、评估方法及当前挑战的解决方案。

Comments Accepted at ACM Computing Surveys. 35 pages, 5 figures, 4 tables. Paper List: github.com/xie-lab-ml/awesome-alignment-of-diffusion-models

Efficient Training of Robust Traditional Chinese LLaMA-1B on a Single Consumer GPU: Continual Pre-training, SFT, and DPO

Yu-Cheng Chih, Ming-Tao Duan, Yong-Hao Hou

专题命中 偏好对齐 ：DPO(title,abstract)；分类 cs.CL

Comments 17 pages, 1 figures, 2 tables. Technical report. Introduces PureTC-1B, an adapter-based pipeline for stabilizing Small Language Models in Traditional Chinese using CPT, SFT, and DPO

Data-Centric Human Preference with Rationales for Direct Preference Alignment

Hoang Anh Just, Ming Jin, Anit Sahu, Huy Phan, Ruoxi Jia

机构 * Virginia Tech（弗吉尼亚理工大学） ； Amazon（亚马逊公司） 纠错

专题命中 偏好对齐 ：alignment(title,abstract)；分类 cs.LG

Comments Data-Centric Human Preference with Rationales for Direct Preference Alignment