大模型对齐与安全 - arXivDaily 专题

2411.16502 2025-02-27 cs.LG cs.AI 73%

Interpreting Language Reward Models via Contrastive Explanations

Junqi Jiang, Tom Bewley, Saumitra Mishra, Freddy Lecue, Manuela Veloso

专题命中偏好对齐：alignment(abstract)；trustworthy(abstract)；分类 cs.AI、cs.LG

Comments Accepted at ICLR 2025 conference

2502.18532 2025-02-27 cs.AI cs.LG 73%

CuDIP: Enhancing Theorem Proving in LLMs via Curriculum Learning-based Direct Preference Optimization

Shuming Shi, Ruobing Zuo, Gaolei He, Jianlin Wang, Chenyang Xu, Zhengfeng Yang

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2502.18480 2025-02-27 cs.IR cs.AI cs.CL 73%

QExplorer: Large Language Model Based Query Extraction for Toxic Content Exploration

Shaola Ren, Li Ke, Longtao Huang, Dehong Gao, Hui Xue

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2502.17515 2025-02-26 cs.LG cs.AI 73%

Towards User-level Private Reinforcement Learning with Human Feedback

Jiaming Zhang, Mingxi Lei, Meng Ding, Mengdi Li, Zihang Xiang, Difei Xu, Jinhui Xu, Di Wang

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2502.12066 2025-02-18 cs.AI cs.LG cs.SE 73%

CONSTRUCTA: Automating Commercial Construction Schedules in Fabrication Facilities with Large Language Models

Yifan Zhang, Xue Yang

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2410.21662 2025-02-18 cs.CL cs.LG 73%

$f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization

Jiaqi Han, Mingjian Jiang, Yuxuan Song, Stefano Ermon, Minkai Xu

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Journal ref AISTATS 2025

2502.06869 2025-02-12 cs.LG cs.AI 73%

A Survey on Explainable Deep Reinforcement Learning

Zelei Cheng, Jiahao Yu, Xinyu Xing

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2502.06861 2025-02-12 cs.LG cs.AI 73%

Design Considerations in Offline Preference-based RL

Alekh Agarwal, Christoph Dann, Teodor V. Marinov

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2410.02205 2025-02-11 cs.CL cs.AI cs.LO 73%

Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models

Yinhong Liu, Zhijiang Guo, Tianya Liang, Ehsan Shareghi, Ivan Vulić, Nigel Collier

专题命中偏好对齐：alignment(abstract)；trustworthy(abstract)；分类 cs.CL、cs.AI

2501.13927 2025-01-24 cs.CL cs.AI cs.CV 73%

CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation

Guofeng Cui, Pichao Wang, Yang Liu, Zemian Ke, Zhu Liu, Vimal Bhat

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2501.11621 2025-01-22 cs.CL cs.LG 73%

Trojan Detection Through Pattern Recognition for Large Language Models

Vedant Bhasin, Matthew Yudin, Razvan Stefanescu, Rauf Izmailov

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments 20 pages, 11 Figures

2412.15838 2024-12-31 cs.AI cs.CL 73%

Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback

Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong, Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan, Jiahao Li, Mohan Wang, Josef Dai, Tianyi Qiu, Hua Xu, Dong Li, Weipeng Chen, Jun Song, Bo Zheng, Yaodong Yang

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

2403.09032 2024-12-30 cs.SE cs.CL cs.LG 73%

CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences

Martin Weyssow, Aton Kamanda, Xin Zhou, Houari Sahraoui

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2402.14228 2024-12-24 cs.LG cs.AI 73%

COPR: Continual Human Preference Learning via Optimal Policy Regularization

Han Zhang, Lin Gui, Yu Lei, Yuanzhao Zhai, Yehong Zhang, Yulan He, Hui Wang, Yue Yu, Kam-Fai Wong, Bin Liang, Ruifeng Xu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments This is a duplicate submission to arXiv:2310.15694, and we believe that this submission has affected the citation of our original paper arXiv:2310.15694

2412.08347 2024-12-12 cs.CL cs.AI 73%

SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs

Sultan Alrashed

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments 10 pages, 4 figures, and 13 tables. For the SmolTulu-1.7b-instruct model, see: https://huggingface.co/SultanR/SmolTulu-1.7b-Instruct

2411.02461 2024-11-06 cs.CL cs.AI 73%

Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Control

Yuxin Xiao, Chaoqun Wan, Yonggang Zhang, Wenxiao Wang, Binbin Lin, Xiaofei He, Xu Shen, Jieping Ye

专题命中偏好对齐：RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI

2405.14734 2024-11-05 cs.CL cs.LG 73%

SimPO: Simple Preference Optimization with a Reference-Free Reward

Yu Meng, Mengzhou Xia, Danqi Chen

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments NeurIPS 2024. Code & models: https://github.com/princeton-nlp/SimPO. v3 updates: Gemma 2 results (Appendix J); more discussions about length normalization (Section 2.2) and KL regularization (Section 2.3)

2410.20298 2024-10-29 cs.CL cs.AI 73%

Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data

Xinhong Xie, Tao Li, Quanyan Zhu

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2410.20187 2024-10-29 cs.LG cs.AI stat.ML 73%

Uncertainty-Penalized Direct Preference Optimization

Sam Houliston, Alizée Pace, Alexander Immer, Gunnar Rätsch

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

Comments Accepted at the NeurIPS 2024 FITML Workshop

2410.18585 2024-10-25 cs.AI cs.LG 73%

Aligning CodeLLMs with Direct Preference Optimization

Yibo Miao, Bofei Gao, Shanghaoran Quan, Junyang Lin, Daoguang Zan, Jiaheng Liu, Jian Yang, Tianyu Liu, Zhijie Deng

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2410.08134 2024-10-11 cs.LG cs.AI 73%

Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction

Jarrid Rector-Brooks, Mohsin Hasan, Zhangzhi Peng, Zachary Quinn, Chenghao Liu, Sarthak Mittal, Nouha Dziri, Michael Bronstein, Yoshua Bengio, Pranam Chatterjee, Alexander Tong, Avishek Joey Bose

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2410.05328 2024-10-10 cs.LG cs.AI 73%

Reward Learning From Preference With Ties

Jinsong Liu, Dongdong Ge, Ruihao Zhu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

2403.19270 2024-10-08 cs.CL cs.AI 73%

sDPO: Don't Use Your Data All at Once

Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, Chanjun Park

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2406.10858 2024-09-30 cs.CL cs.AI 73%

Step-level Value Preference Optimization for Mathematical Reasoning

Guoxin Chen, Minpeng Liao, Chengxi Li, Kai Fan

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Camera ready version for EMNLP2024-Findings

2406.09920 2024-09-25 cs.CL cs.AI 73%

Knowledge Editing in Language Models via Adapted Direct Preference Optimization

Amit Rozner, Barak Battash, Lior Wolf, Ofir Lindenbaum

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments 9 pages, 4 figures

2408.16751 2024-08-30 cs.CL cs.LG stat.ML 73%

A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models

Yi-Lin Tuan, William Yang Wang

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2408.10642 2024-08-21 cs.AI cs.CL 73%

Minor SFT loss for LLM fine-tune to increase performance and reduce model deviation

Shiming Xie, Hong Chen, Fred Yu, Zeye Sun, Xiuyu Wu

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments 8 pages, 5 figures

2408.07055 2024-08-14 cs.CL cs.LG 73%

LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs

Yushi Bai, Jiajie Zhang, Xin Lv, Linzhi Zheng, Siqi Zhu, Lei Hou, Yuxiao Dong, Jie Tang, Juanzi Li

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2402.00742 2024-07-22 cs.CL cs.AI 73%

Transforming and Combining Rewards for Aligning Large Language Models

Zihao Wang, Chirag Nagpal, Jonathan Berant, Jacob Eisenstein, Alex D'Amour, Sanmi Koyejo, Victor Veitch

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

2407.04842 2024-07-09 cs.CV cs.CL cs.LG 73%

MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?

Zhaorun Chen, Yichao Du, Zichen Wen, Yiyang Zhou, Chenhang Cui, Zhenzhen Weng, Haoqin Tu, Chaoqi Wang, Zhengwei Tong, Qinglan Huang, Canyu Chen, Qinghao Ye, Zhihong Zhu, Yuqing Zhang, Jiawei Zhou, Zhuokai Zhao, Rafael Rafailov, Chelsea Finn, Huaxiu Yao

专题命中偏好对齐：alignment(abstract)；safety(abstract)；分类 cs.CL、cs.LG

Comments 42 pages, 13 figures, 33 tables

1. 偏好对齐 3235 篇

Interpreting Language Reward Models via Contrastive Explanations

CuDIP: Enhancing Theorem Proving in LLMs via Curriculum Learning-based Direct Preference Optimization

QExplorer: Large Language Model Based Query Extraction for Toxic Content Exploration

Towards User-level Private Reinforcement Learning with Human Feedback

CONSTRUCTA: Automating Commercial Construction Schedules in Fabrication Facilities with Large Language Models

$f$-PO: Generalizing Preference Optimization with $f$-divergence Minimization

A Survey on Explainable Deep Reinforcement Learning

Design Considerations in Offline Preference-based RL

Aligning with Logic: Measuring, Evaluating and Improving Logical Preference Consistency in Large Language Models

CRPO: Confidence-Reward Driven Preference Optimization for Machine Translation

Trojan Detection Through Pattern Recognition for Large Language Models

Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback

CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferences

COPR: Continual Human Preference Learning via Optimal Policy Regularization

SmolTulu: Higher Learning Rate to Batch Size Ratios Can Lead to Better Reasoning in SLMs

Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Control

SimPO: Simple Preference Optimization with a Reference-Free Reward

Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data

Uncertainty-Penalized Direct Preference Optimization

Aligning CodeLLMs with Direct Preference Optimization

Steering Masked Discrete Diffusion Models via Discrete Denoising Posterior Prediction

Reward Learning From Preference With Ties

sDPO: Don't Use Your Data All at Once

Step-level Value Preference Optimization for Mathematical Reasoning

Knowledge Editing in Language Models via Adapted Direct Preference Optimization

A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models

Minor SFT loss for LLM fine-tune to increase performance and reduce model deviation

LongWriter: Unleashing 10,000+ Word Generation from Long Context LLMs

Transforming and Combining Rewards for Aligning Large Language Models

MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?