大模型对齐与安全 - arXivDaily 专题

2505.14946 2025-05-22 cs.AI 70%

Reinforcement Learning from User Feedback

Eric Han, Jun Chen, Karthik Abinav Sankararaman, Xiaoliang Peng, Tengyu Xu, Eryk Helenowski, Kaiyan Peng, Mrinal Kumar, Sinong Wang, Han Fang, Arya Talebzadeh

机构 * Meta GenAI（Meta 生成式人工智能）

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.AI

2412.20299 2025-05-14 cs.CL 70%

No Preference Left Behind: Group Distributional Preference Optimization

Binwei Yao, Zefan Cai, Yun-Shiuan Chuang, Shanglin Yang, Ming Jiang, Diyi Yang, Junjie Hu

机构 * University of Wisconsin-Madison（威斯康星大学麦迪逊分校）； Indiana University Indianapolis（印第安纳大学印第安纳波利斯分校）； Stanford University（斯坦福大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

2411.03700 2025-05-13 cs.CL 70%

The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models

Anaelia Ovalle, Krunoslav Lehman Pavasovic, Louis Martin, Luke Zettlemoyer, Eric Michael Smith, Kai-Wei Chang, Adina Williams, Levent Sagun

机构 * FAIR, Meta（FAIR与Meta）； Meta United States（Meta美国分公司）； Meta France（Meta法国分公司）； Gen AI, Meta（Meta生成式人工智能部门）； UCLA United States（加州大学洛杉矶分校）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments Accepted to 2025 ACM FAccT

2505.04993 2025-05-09 cs.CL 70%

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

Zhuocheng Gong, Jian Guan, Wei Wu, Huishuai Zhang, Dongyan Zhao

机构 * Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale Department of Computer Science Cranberry-Lemon University Pittsburgh, PA 15213, USA（计算机科学系， Cranberry-Lemon 大学）； Ji Q. Ren & Yevgeny LeNet Department of Computational Neuroscience University of the Witwatersrand Joburg, South Africa（计算神经科学系，威特沃特斯兰大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

2505.02363 2025-05-06 cs.CL 70%

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning

Tianjian Li, Daniel Khashabi

机构 * Johns Hopkins University（约翰霍普金斯大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments To appear in ICML 2025

2504.16272 2025-04-24 cs.LG 70%

Learning Explainable Dense Reward Shapes via Bayesian Optimization

Ryan Koo, Ian Yang, Vipul Raheja, Mingyi Hong, Kwang-Sung Jun, Dongyeop Kang

机构 * University of Minnesota（明尼苏达大学）； Georgia Institute of Technology（佐治亚理工学院）； Grammarly ； University of Arizona（亚利桑那大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

2502.10482 2025-04-18 cs.AI 70%

A Self-Supervised Reinforcement Learning Approach for Fine-Tuning Large Language Models Using Cross-Attention Signals

Andrew Kiruluta, Andreas Lemos, Priscilla Burity

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI

2504.06141 2025-04-15 cs.LG 70%

Adversarial Training of Reward Models

Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksii Kuchaiev, Olivier Delalleau, Tuo Zhao

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

Comments 16 pages, 7 figures

2404.16020 2025-04-10 cs.CL 70%

Investigating Adversarial Trigger Transfer in Large Language Models

Nicholas Meade, Arkil Patel, Siva Reddy

专题命中偏好对齐：safety(abstract)；jailbreak(abstract)；分类 cs.CL

2411.15382 2025-04-01 cs.CL 70%

On the Impact of Fine-Tuning on Chain-of-Thought Reasoning

Elita Lobo, Chirag Agarwal, Himabindu Lakkaraju

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.CL

Comments This paper is a work in progress with findings based on limited evidence. Please exercise discretion when interpreting the findings

2503.15129 2025-03-20 cs.AI 70%

Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Models

Man Fai Wong, Chee Wei Tan

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI

2503.14391 2025-03-19 cs.CL 70%

How much do LLMs learn from negative examples?

Shadi Hamdan, Deniz Yuret

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments 8 pages, 6 figures

2503.14189 2025-03-19 cs.CL cs.CV 70%

Towards Harmless Multimodal Assistants with Blind Preference Optimization

Yongqi Li, Lu Yang, Jian Wang, Runyang You, Wenjie Li, Liqiang Nie

专题命中偏好对齐：DPO(abstract)；safety(abstract)；分类 cs.CL

2503.10215 2025-03-14 cs.AI cs.GT 70%

Adaptive Preference Aggregation

Benjamin Heymann

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI

2503.01076 2025-03-04 cs.LG stat.ML 70%

Active Learning for Direct Preference Optimization

Branislav Kveton, Xintong Li, Julian McAuley, Ryan Rossi, Jingbo Shang, Junda Wu, Tong Yu

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.LG

2409.17401 2025-03-04 cs.LG stat.ML 70%

Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference

Qining Zhang, Lei Ying

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.LG

2502.19798 2025-02-28 cs.AI 70%

Developmental Support Approach to AI's Autonomous Growth: Toward the Realization of a Mutually Beneficial Stage Through Experiential Learning

Taichiro Endo

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

Comments 4pages, 3 figures

Journal ref Proc. 1st Workshop on Post-Singularity Symbiosis, PSS-2025-007, March 3, 2025

2408.07471 2025-02-19 cs.CL 70%

Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization

Yuxin Jiang, Bo Huang, Yufei Wang, Xingshan Zeng, Liangyou Li, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Wei Wang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments 20 pages, 9 figures, 12 tables. Accepted at ICLR 2025

2407.02302 2025-02-19 cs.CL 70%

Towards Human Understanding of Paraphrase Types in Large Language Models

Dominik Meier, Jan Philip Wahle, Terry Ruas, Bela Gipp

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL

Journal ref Proceedings of the 31st International Conference on Computational Linguistics (2025), pages 6298-6316

2410.21545 2025-02-19 cs.CL 70%

CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling

Taneesh Gupta, Shivam Shandilya, Xuchao Zhang, Rahul Madhavan, Supriyo Ghosh, Chetan Bansal, Huaxiu Yao, Saravan Rajmohan

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL

2410.00847 2025-02-13 cs.LG 70%

Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown

Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, Junge Zhang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.LG

2407.08770 2025-02-12 cs.AI 70%

Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing

Huanqian Wang, Yang Yue, Rui Lu, Jingxin Shi, Andrew Zhao, Shenzhi Wang, Shiji Song, Gao Huang

专题命中偏好对齐：RLHF(abstract)；jailbreak(abstract)；分类 cs.AI

Comments 23 pages, 14 figures

2501.18457 2025-02-11 cs.CL 70%

CALM: Unleashing the Cross-Lingual Self-Aligning Ability of Language Model Question Answering

Yumeng Wang, Zhiyuan Fan, Qingyun Wang, May Fung, Heng Ji

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments Accepted by NAACL 2025

2501.18663 2025-02-03 cs.CR cs.AI 70%

Joint Optimization of Prompt Security and System Performance in Edge-Cloud LLM Systems

Haiyang Huang, Tianhui Meng, Weijia Jia

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.AI

2407.01158 2025-01-28 cs.CL 70%

Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation

Takyoung Kim, Kyungjae Lee, Young Rok Jang, Ji Yong Cho, Gangwoo Kim, Minseok Cho, Moontae Lee

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments NAACL 2025 (Findings, Long). Resources are available at https://github.com/youngerous/qtree

2501.07888 2025-01-27 cs.CV cs.AI 70%

Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding

Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, Yuan Lin

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

2312.14436 2025-01-22 cs.RO cs.LG 70%

REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback

Souradip Chakraborty, Anukriti Singh, Amisha Bhaskar, Pratap Tokekar, Dinesh Manocha, Amrit Singh Bedi

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

2501.01336 2025-01-03 cs.CL 70%

Aligning Large Language Models for Faithful Integrity Against Opposing Argument

Yong Zhao, Yang Deng, See-Kiong Ng, Tat-Seng Chua

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments 17 pages, 5 figures

2410.02504 2025-01-03 stat.ML cs.LG 70%

Dual Active Learning for Reinforcement Learning from Human Feedback

Pangpang Liu, Chengchun Shi, Will Wei Sun

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

2412.15650 2024-12-23 cs.LG 70%

Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution

Wentao Tan, Qiong Cao, Yibing Zhan, Chao Xue, Changxing Ding

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.LG

Comments AAAI 2025. The code is available at https://github.com/WentaoTan/SENA

1. 偏好对齐 3243 篇

Reinforcement Learning from User Feedback

No Preference Left Behind: Group Distributional Preference Optimization

The Root Shapes the Fruit: On the Persistence of Gender-Exclusive Harms in Aligned Language Models

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning

Learning Explainable Dense Reward Shapes via Bayesian Optimization

A Self-Supervised Reinforcement Learning Approach for Fine-Tuning Large Language Models Using Cross-Attention Signals

Adversarial Training of Reward Models

Investigating Adversarial Trigger Transfer in Large Language Models

On the Impact of Fine-Tuning on Chain-of-Thought Reasoning

Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Models

How much do LLMs learn from negative examples?

Towards Harmless Multimodal Assistants with Blind Preference Optimization

Adaptive Preference Aggregation

Active Learning for Direct Preference Optimization

Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference

Developmental Support Approach to AI's Autonomous Growth: Toward the Realization of a Mutually Beneficial Stage Through Experiential Learning

Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization

Towards Human Understanding of Paraphrase Types in Large Language Models

CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling

Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown

Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing

CALM: Unleashing the Cross-Lingual Self-Aligning Ability of Language Model Question Answering

Joint Optimization of Prompt Security and System Performance in Edge-Cloud LLM Systems

Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation

Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding

REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback

Aligning Large Language Models for Faithful Integrity Against Opposing Argument

Dual Active Learning for Reinforcement Learning from Human Feedback

Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution