大模型对齐与安全 - arXivDaily 专题

2505.04993 2025-05-09 cs.CL 70%

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

Zhuocheng Gong, Jian Guan, Wei Wu, Huishuai Zhang, Dongyan Zhao

机构 * Antiquus S. Hippocampus, Natalia Cerebro & Amelie P. Amygdale Department of Computer Science Cranberry-Lemon University Pittsburgh, PA 15213, USA（计算机科学系， Cranberry-Lemon 大学）； Ji Q. Ren & Yevgeny LeNet Department of Computational Neuroscience University of the Witwatersrand Joburg, South Africa（计算神经科学系，威特沃特斯兰大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

2505.02363 2025-05-06 cs.CL 70%

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning

Tianjian Li, Daniel Khashabi

机构 * Johns Hopkins University（约翰霍普金斯大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments To appear in ICML 2025

2504.16272 2025-04-24 cs.LG 70%

Learning Explainable Dense Reward Shapes via Bayesian Optimization

Ryan Koo, Ian Yang, Vipul Raheja, Mingyi Hong, Kwang-Sung Jun, Dongyeop Kang

机构 * University of Minnesota（明尼苏达大学）； Georgia Institute of Technology（佐治亚理工学院）； Grammarly ； University of Arizona（亚利桑那大学）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

2502.10482 2025-04-18 cs.AI 70%

A Self-Supervised Reinforcement Learning Approach for Fine-Tuning Large Language Models Using Cross-Attention Signals

Andrew Kiruluta, Andreas Lemos, Priscilla Burity

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI

2504.06141 2025-04-15 cs.LG 70%

Adversarial Training of Reward Models

Alexander Bukharin, Haifeng Qian, Shengyang Sun, Adithya Renduchintala, Soumye Singhal, Zhilin Wang, Oleksii Kuchaiev, Olivier Delalleau, Tuo Zhao

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

Comments 16 pages, 7 figures

2404.16020 2025-04-10 cs.CL 70%

Investigating Adversarial Trigger Transfer in Large Language Models

Nicholas Meade, Arkil Patel, Siva Reddy

专题命中偏好对齐：safety(abstract)；jailbreak(abstract)；分类 cs.CL

2411.15382 2025-04-01 cs.CL 70%

On the Impact of Fine-Tuning on Chain-of-Thought Reasoning

Elita Lobo, Chirag Agarwal, Himabindu Lakkaraju

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.CL

Comments This paper is a work in progress with findings based on limited evidence. Please exercise discretion when interpreting the findings

2503.15129 2025-03-20 cs.AI 70%

Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Models

Man Fai Wong, Chee Wei Tan

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI

2503.14391 2025-03-19 cs.CL 70%

How much do LLMs learn from negative examples?

Shadi Hamdan, Deniz Yuret

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL

Comments 8 pages, 6 figures

2503.14189 2025-03-19 cs.CL cs.CV 70%

Towards Harmless Multimodal Assistants with Blind Preference Optimization

Yongqi Li, Lu Yang, Jian Wang, Runyang You, Wenjie Li, Liqiang Nie

专题命中偏好对齐：DPO(abstract)；safety(abstract)；分类 cs.CL

2503.10215 2025-03-14 cs.AI cs.GT 70%

Adaptive Preference Aggregation

Benjamin Heymann

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI

2503.01076 2025-03-04 cs.LG stat.ML 70%

Active Learning for Direct Preference Optimization

Branislav Kveton, Xintong Li, Julian McAuley, Ryan Rossi, Jingbo Shang, Junda Wu, Tong Yu

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.LG

2409.17401 2025-03-04 cs.LG stat.ML 70%

Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference

Qining Zhang, Lei Ying

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.LG

2502.19798 2025-02-28 cs.AI 70%

Developmental Support Approach to AI's Autonomous Growth: Toward the Realization of a Mutually Beneficial Stage Through Experiential Learning

Taichiro Endo

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

Comments 4pages, 3 figures

Journal ref Proc. 1st Workshop on Post-Singularity Symbiosis, PSS-2025-007, March 3, 2025

2408.07471 2025-02-19 cs.CL 70%

Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization

Yuxin Jiang, Bo Huang, Yufei Wang, Xingshan Zeng, Liangyou Li, Yasheng Wang, Xin Jiang, Lifeng Shang, Ruiming Tang, Wei Wang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments 20 pages, 9 figures, 12 tables. Accepted at ICLR 2025

2407.02302 2025-02-19 cs.CL 70%

Towards Human Understanding of Paraphrase Types in Large Language Models

Dominik Meier, Jan Philip Wahle, Terry Ruas, Bela Gipp

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL

Journal ref Proceedings of the 31st International Conference on Computational Linguistics (2025), pages 6298-6316

2410.21545 2025-02-19 cs.CL 70%

CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling

Taneesh Gupta, Shivam Shandilya, Xuchao Zhang, Rahul Madhavan, Supriyo Ghosh, Chetan Bansal, Huaxiu Yao, Saravan Rajmohan

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL

2410.00847 2025-02-13 cs.LG 70%

Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown

Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, Junge Zhang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.LG

2407.08770 2025-02-12 cs.AI 70%

Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing

Huanqian Wang, Yang Yue, Rui Lu, Jingxin Shi, Andrew Zhao, Shenzhi Wang, Shiji Song, Gao Huang

专题命中偏好对齐：RLHF(abstract)；jailbreak(abstract)；分类 cs.AI

Comments 23 pages, 14 figures

2501.18457 2025-02-11 cs.CL 70%

CALM: Unleashing the Cross-Lingual Self-Aligning Ability of Language Model Question Answering

Yumeng Wang, Zhiyuan Fan, Qingyun Wang, May Fung, Heng Ji

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments Accepted by NAACL 2025

2501.18663 2025-02-03 cs.CR cs.AI 70%

Joint Optimization of Prompt Security and System Performance in Edge-Cloud LLM Systems

Haiyang Huang, Tianhui Meng, Weijia Jia

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.AI

2407.01158 2025-01-28 cs.CL 70%

Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation

Takyoung Kim, Kyungjae Lee, Young Rok Jang, Ji Yong Cho, Gangwoo Kim, Minseok Cho, Moontae Lee

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments NAACL 2025 (Findings, Long). Resources are available at https://github.com/youngerous/qtree

2501.07888 2025-01-27 cs.CV cs.AI 70%

Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding

Liping Yuan, Jiawei Wang, Haomiao Sun, Yuchen Zhang, Yuan Lin

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI

2312.14436 2025-01-22 cs.RO cs.LG 70%

REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback

Souradip Chakraborty, Anukriti Singh, Amisha Bhaskar, Pratap Tokekar, Dinesh Manocha, Amrit Singh Bedi

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

2501.01336 2025-01-03 cs.CL 70%

Aligning Large Language Models for Faithful Integrity Against Opposing Argument

Yong Zhao, Yang Deng, See-Kiong Ng, Tat-Seng Chua

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments 17 pages, 5 figures

2410.02504 2025-01-03 stat.ML cs.LG 70%

Dual Active Learning for Reinforcement Learning from Human Feedback

Pangpang Liu, Chengchun Shi, Will Wei Sun

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.LG

2412.15650 2024-12-23 cs.LG 70%

Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution

Wentao Tan, Qiong Cao, Yibing Zhan, Chao Xue, Changxing Ding

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.LG

Comments AAAI 2025. The code is available at https://github.com/WentaoTan/SENA

2412.15156 2024-12-20 cs.CV cs.CL cs.MM 70%

Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM

Yatai Ji, Jiacheng Zhang, Jie Wu, Shilong Zhang, Shoufa Chen, Chongjian GE, Peize Sun, Weifeng Chen, Wenqi Shao, Xuefeng Xiao, Weilin Huang, Ping Luo

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

2402.01766 2024-12-20 cs.CL cs.AI cs.CY cs.LG econ.GN q-fin.EC 70%

LLM Voting: Human Choices and AI Collective Decision Making

Joshua C. Yang, Damian Dailisan, Marcin Korecki, Carina I. Hausladen, Dirk Helbing

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.CY

Comments Accepted in AAAI Conference on AI, Ethics, and Society (AIES)

2412.12865 2024-12-18 cs.CL 70%

Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models

Yuchen Fan, Yuzhong Hong, Qiushi Wang, Junwei Bao, Hongfei Jiang, Yang Song

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL

Comments AAAI2025, 12 pages, 9 figures

1. 偏好对齐 3238 篇

Latent Preference Coding: Aligning Large Language Models via Discrete Latent Codes

SIMPLEMIX: Frustratingly Simple Mixing of Off- and On-policy Data in Language Model Preference Learning

Learning Explainable Dense Reward Shapes via Bayesian Optimization

A Self-Supervised Reinforcement Learning Approach for Fine-Tuning Large Language Models Using Cross-Attention Signals

Adversarial Training of Reward Models

Investigating Adversarial Trigger Transfer in Large Language Models

On the Impact of Fine-Tuning on Chain-of-Thought Reasoning

Aligning Crowd-sourced Human Feedback for Reinforcement Learning on Code Generation by Large Language Models

How much do LLMs learn from negative examples?

Towards Harmless Multimodal Assistants with Blind Preference Optimization

Adaptive Preference Aggregation

Active Learning for Direct Preference Optimization

Zeroth-Order Policy Gradient for Reinforcement Learning from Human Feedback without Reward Inference

Developmental Support Approach to AI's Autonomous Growth: Toward the Realization of a Mutually Beneficial Stage Through Experiential Learning

Bridging and Modeling Correlations in Pairwise Data for Direct Preference Optimization

Towards Human Understanding of Paraphrase Types in Large Language Models

CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling

Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown

Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editing

CALM: Unleashing the Cross-Lingual Self-Aligning Ability of Language Model Question Answering

Joint Optimization of Prompt Security and System Performance in Edge-Cloud LLM Systems

Learning to Explore and Select for Coverage-Conditioned Retrieval-Augmented Generation

Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding

REBEL: Reward Regularization-Based Approach for Robotic Reinforcement Learning from Human Feedback

Aligning Large Language Models for Faithful Integrity Against Opposing Argument

Dual Active Learning for Reinforcement Learning from Human Feedback

Beyond Human Data: Aligning Multimodal Large Language Models by Iterative Self-Evolution

Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLM

LLM Voting: Human Choices and AI Collective Decision Making

Preference-Oriented Supervised Fine-Tuning: Favoring Target Model Over Aligned Large Language Models