大模型对齐与安全 - arXivDaily 专题

2411.05875 2024-11-12 cs.LG cs.AI cs.CL 75%

Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization

Zhuotong Chen, Fang Liu, Jennifer Zhu, Wanyu Du, Yanjun Qi

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 15 pages

2406.16235 2024-11-11 cs.CL cs.AI cs.CR cs.LG 75%

Preference Tuning For Toxicity Mitigation Generalizes Across Languages

Xiaochen Li, Zheng-Xin Yong, Stephen H. Bach

专题命中偏好对齐：DPO(abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Findings of EMNLP 2024

2410.22891 2024-10-31 cs.LG cs.AI cs.CL 75%

VPO: Leveraging the Number of Votes in Preference Optimization

Jae Hyeon Cho, Minkyung Park, Byung-Jun Lee

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2401.12086 2024-10-28 cs.CL cs.AI cs.LG 75%

West-of-N: Synthetic Preferences for Self-Improving Reward Models

Alizée Pace, Jonathan Mallinson, Eric Malmi, Sebastian Krause, Aliaksei Severyn

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2401.16635 2024-10-23 cs.LG cs.AI cs.CL 75%

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, Chuang Gan

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2405.19567 2024-10-11 cs.AI cs.CL cs.CV cs.LG 75%

Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding

Shenghuan Sun, Alexander Schubert, Gregory M. Goldgof, Zhiqing Sun, Thomas Hartvigsen, Atul J. Butte, Ahmed Alaa

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Code available at: https://github.com/AlaaLab/Dr-LLaVA

2406.11839 2024-10-08 cs.CV cs.AI cs.CL cs.LG 75%

mDPO: Conditional Preference Optimization for Multimodal Large Language Models

Fei Wang, Wenxuan Zhou, James Y. Huang, Nan Xu, Sheng Zhang, Hoifung Poon, Muhao Chen

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to EMNLP 2024 Main Conference. Project website: https://feiwang96.github.io/mDPO

2403.10704 2024-09-16 cs.LG cs.AI cs.CL 75%

Parameter Efficient Reinforcement Learning from Human Feedback

Hakim Sidahmed, Samrat Phatale, Alex Hutcheson, Zhuonan Lin, Zhang Chen, Zac Yu, Jarvis Jin, Simral Chaudhary, Roman Komarytsia, Christiane Ahlheim, Yonghao Zhu, Bowen Li, Saravanan Ganesh, Bill Byrne, Jessica Hoffmann, Hassan Mansoor, Wei Li, Abhinav Rastogi, Lucas Dixon

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2409.02118 2024-09-05 cs.LG cs.AI cs.CL 75%

TSO: Self-Training with Scaled Preference Optimization

Kaihui Chen, Hao Yi, Qingyang Li, Tianyu Qi, Yulan Hu, Fuzheng Zhang, Yong Liu

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2408.10075 2024-08-20 cs.LG cs.AI cs.CL cs.RO 75%

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

Sriyash Poddar, Yanming Wan, Hamish Ivison, Abhishek Gupta, Natasha Jaques

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments weirdlabuw.github.io/vpl

2305.18290 2024-07-31 cs.LG cs.AI cs.CL 75%

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2404.08699 2024-07-30 cs.CL cs.AI cs.LG 75%

PoliTune: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in Large Language Models

Ahmed Agiza, Mohamed Mostagir, Sherief Reda

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments AIES '24: Proceedings of the 2024 AAAI/ACM Conference on AI, Ethics, and Society

2407.14622 2024-07-23 cs.LG cs.AI cs.CL 75%

BOND: Aligning LLMs with Best-of-N Distillation

Pier Giuseppe Sessa, Robert Dadashi, Léonard Hussenot, Johan Ferret, Nino Vieillard, Alexandre Ramé, Bobak Shariari, Sarah Perrin, Abe Friesen, Geoffrey Cideron, Sertan Girgin, Piotr Stanczyk, Andrea Michi, Danila Sinopalnikov, Sabela Ramos, Amélie Héliou, Aliaksei Severyn, Matt Hoffman, Nikola Momchev, Olivier Bachem

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

2407.02119 2024-07-10 cs.LG cs.AI cs.CL 75%

Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning

Yifang Chen, Shuohang Wang, Ziyi Yang, Hiteshi Sharma, Nikos Karampatziakis, Donghan Yu, Kevin Jamieson, Simon Shaolei Du, Yelong Shen

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.08114 2024-07-01 cs.LG cs.AI cs.CL 75%

Active Preference Learning for Large Language Models

William Muldrew, Peter Hayes, Mingtian Zhang, David Barber

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 13 pages, 5 figures, 6 tables

2406.08673 2024-06-14 cs.CL cs.AI cs.LG 75%

HelpSteer2: Open-source dataset for training top-performing reward models

Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, Oleksii Kuchaiev

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.02756 2024-06-06 cs.CL cs.AI cs.LG 75%

Aligning Large Language Models via Fine-grained Supervision

Dehong Xu, Liang Qiu, Minseok Kim, Faisal Ladhak, Jaeyoung Do

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.02992 2024-05-27 cs.LG cs.AI cs.CL 75%

Decoding-time Realignment of Language Models

Tianlin Liu, Shangmin Guo, Leonardo Bianco, Daniele Calandriello, Quentin Berthet, Felipe Llinares, Jessica Hoffmann, Lucas Dixon, Michal Valko, Mathieu Blondel

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments In Proceedings of the 41st International Conference on Machine Learning (ICML 2024)

2310.02263 2024-04-04 cs.CL cs.AI cs.LG 75%

Automatic Pair Construction for Contrastive Post-training

Canwen Xu, Corby Rosset, Ethan C. Chau, Luciano Del Corro, Shweti Mahajan, Julian McAuley, Jennifer Neville, Ahmed Hassan Awadallah, Nikhil Rao

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NAACL 2024 (Findings)

2403.10949 2024-03-27 cs.CL cs.AI cs.LG 75%

SelfIE: Self-Interpretation of Large Language Model Embeddings

Haozhe Chen, Carl Vondrick, Chengzhi Mao

专题命中偏好对齐：RLHF(abstract)；prompt injection(abstract)；分类 cs.CL、cs.AI、cs.LG

2403.06754 2024-03-19 cs.CL cs.AI cs.LG 75%

ALaRM: Align Language Models via Hierarchical Rewards Modeling

Yuhang Lai, Siyuan Wang, Shujun Liu, Xuanjing Huang, Zhongyu Wei

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 15 pages, 6 figures

2305.06176 2024-03-06 cs.CL cs.AI cs.LG 75%

Fine-tuning Language Models with Generative Adversarial Reward Modelling

Zhang Ze Yu, Lau Jia Jaw, Zhang Hui, Bryan Kian Hsiang Low

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 22 pages, 9 figures, 12 tables

2402.10210 2024-02-16 cs.LG cs.AI cs.CL cs.CV stat.ML 75%

Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation

Huizhuo Yuan, Zixiang Chen, Kaixuan Ji, Quanquan Gu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 28 pages, 8 figures, 10 tables

2401.12187 2024-01-23 cs.LG cs.AI cs.CL 75%

WARM: On the Benefits of Weight Averaged Reward Models

Alexandre Ramé, Nino Vieillard, Léonard Hussenot, Robert Dadashi, Geoffrey Cideron, Olivier Bachem, Johan Ferret

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 14 pages, 9 figures

2305.14387 2024-01-09 cs.LG cs.AI cs.CL 75%

AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto

专题命中偏好对齐：DPO(abstract)；trustworthy(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Spotlight at NeurIPS 2023

2305.11455 2023-05-22 cs.CL cs.AI cs.LG 75%

Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models

Wanqiao Xu, Shi Dong, Dilip Arumugam, Benjamin Van Roy

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2408.04638 2025-09-09 cs.CL cs.CY 74%

Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspective

Yiqun Zhang, Xiaocui Yang, Xingle Xu, Zeran Gao, Yijie Huang, Shiyi Mu, Shi Feng, Daling Wang, Yifei Zhang, Kaisong Song, Ge Yu

机构 * Northeastern University, China（东北大学）； Alibaba Group, Hangzhou, China（阿里巴巴集团）

专题命中偏好对齐：RLHF(abstract,comments)；safety(abstract)；分类 cs.CL、cs.CY

Comments Compared with the previous version, reinforcement learning has been added (as a new section), including RLHF, RLVR, and RLAIF

2506.11702 2025-06-16 cs.CL cs.AI 74%

Configurable Preference Tuning with Rubric-Guided Synthetic Data

Víctor Gallego

机构 * Komorebi AI Technologies, Madrid, Spain（Komorebi人工智能技术公司）

专题命中偏好对齐：alignment(abstract,comments)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ICML 2025 Workshop on Models of Human Feedback for AI Alignment

2410.12138 2025-03-27 cs.LG cs.CL 74%

Preference Optimization with Multi-Sample Comparisons

Chaoqi Wang, Zhuokai Zhao, Chen Zhu, Karthik Abinav Sankararaman, Michal Valko, Xuefei Cao, Zhaorun Chen, Madian Khabsa, Yuxin Chen, Hao Ma, Sinong Wang

专题命中偏好对齐：alignment(abstract,comments)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments Code is available at https://github.com/alecwangcq/multi-sample-alignment

2204.05862 2022-04-13 cs.CL cs.LG 74%

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah, Ben Mann, Jared Kaplan

专题命中偏好对齐：RLHF(abstract,comments)；alignment(abstract)；分类 cs.CL、cs.LG

Comments Data available at https://github.com/anthropics/hh-rlhf

1. 偏好对齐 3235 篇

Towards Improved Preference Optimization Pipeline: from Data Generation to Budget-Controlled Regularization

Preference Tuning For Toxicity Mitigation Generalizes Across Languages

VPO: Leveraging the Number of Votes in Preference Optimization

West-of-N: Synthetic Preferences for Self-Improving Reward Models

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding

mDPO: Conditional Preference Optimization for Multimodal Large Language Models

Parameter Efficient Reinforcement Learning from Human Feedback

TSO: Self-Training with Scaled Preference Optimization

Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learning

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

PoliTune: Analyzing the Impact of Data Selection and Fine-Tuning on Economic and Political Biases in Large Language Models

BOND: Aligning LLMs with Best-of-N Distillation

Cost-Effective Proxy Reward Model Construction with On-Policy and Active Learning

Active Preference Learning for Large Language Models

HelpSteer2: Open-source dataset for training top-performing reward models

Aligning Large Language Models via Fine-grained Supervision

Decoding-time Realignment of Language Models

Automatic Pair Construction for Contrastive Post-training

SelfIE: Self-Interpretation of Large Language Model Embeddings

ALaRM: Align Language Models via Hierarchical Rewards Modeling

Fine-tuning Language Models with Generative Adversarial Reward Modelling

Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation

WARM: On the Benefits of Weight Averaged Reward Models

AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback

Shattering the Agent-Environment Interface for Fine-Tuning Inclusive Language Models

Affective Computing in the Era of Large Language Models: A Survey from the NLP Perspective

Configurable Preference Tuning with Rubric-Guided Synthetic Data

Preference Optimization with Multi-Sample Comparisons

Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback