大模型对齐与安全 - arXivDaily 专题

2411.08733 2024-11-15 cs.CL 83%

Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models

Somanshu Singla, Zhen Wang, Tianyang Liu, Abdullah Ashfaq, Zhiting Hu, Eric P. Xing

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments EMNLP 2024 Main

2411.01245 2024-11-05 cs.CL 83%

PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment

Dongxu Liu, Bing Xu, Yinzhuo Chen, Bufan Xu, Wenpeng Lu, Muyun Yang, Tiejun Zhao

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

2405.17888 2024-10-29 cs.AI 83%

Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment

Jiaxiang Li, Siliang Zeng, Hoi-To Wai, Chenliang Li, Alfredo Garcia, Mingyi Hong

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI

2410.09421 2024-10-21 cs.CV cs.CL 83%

VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment

Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, Qi Liu

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.CL

Comments EMNLP 2024 Main Conference camera-ready version (fixed small typos). This article supersedes arXiv:2312.10665

2405.12739 2024-10-14 cs.LG 83%

SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling

Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, Kaiqi Huang

专题命中偏好对齐：alignment(title,abstract)；harmlessness(abstract)；分类 cs.LG

2409.19256 2024-10-03 cs.LG cs.DC 83%

HybridFlow: A Flexible and Efficient RLHF Framework

Guangming Sheng, Chi Zhang, Zilingfeng Ye, Xibin Wu, Wang Zhang, Ru Zhang, Yanghua Peng, Haibin Lin, Chuan Wu

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.LG

2405.01511 2024-08-08 cs.CL 83%

D2PO: Discriminator-Guided DPO with Response Evaluation Models

Prasann Singhal, Nathan Lambert, Scott Niekum, Tanya Goyal, Greg Durrett

专题命中偏好对齐：DPO(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments 20 pages, 12 figures, Accepted to COLM 2024

2407.03051 2024-07-19 cs.CL 83%

Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment

Janghwan Lee, Seongmin Park, Sukjin Hong, Minsoo Kim, Du-Seong Chang, Jungwook Choi

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments ACL 2024 Main

2407.00782 2024-07-16 cs.CL 83%

Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning

Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, Hongsheng Li

专题命中偏好对齐：DPO(title,abstract)；alignment(abstract)；分类 cs.CL

2407.02477 2024-07-03 cs.CV cs.CL 83%

Understanding Alignment in Multimodal LLMs: A Comprehensive Study

Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, Peter Grasch

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL

2401.11458 2024-07-03 cs.CL 83%

Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback

Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, Qi Zhang, Dahua Lin

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments Accepted by ICML2024, I'm still preparing a better vision

2406.19188 2024-06-28 cs.LG 83%

Averaging log-likelihoods in direct alignment

Nathan Grinsztajn, Yannis Flet-Berliac, Mohammad Gheshlaghi Azar, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Olivier Pietquin, Matthieu Geist

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.LG

2406.16316 2024-06-25 cs.CL cs.AI cs.CY cs.LG 83%

Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?

Yuu Jinnai

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.CY

Comments The 2nd Workshop on Cross-Cultural Considerations in NLP (C3NLP) at ACL 2024

2406.08842 2024-06-14 cs.CL 83%

ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions

Xu Zhang, Xunjian Yin, Xiaojun Wan

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL

2402.00856 2024-06-06 cs.CL 83%

Towards Efficient Exact Optimization of Language Model Alignment

Haozhe Ji, Cheng Lu, Yilin Niu, Pei Ke, Hongning Wang, Jun Zhu, Jie Tang, Minlie Huang

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL

Comments 24 pages, 9 figures

Journal ref Forty-first International Conference on Machine Learning (ICML 2024)

2404.10271 2024-06-05 cs.LG cs.AI cs.CL cs.CY cs.GT 83%

Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback

Vincent Conitzer, Rachel Freedman, Jobst Heitzig, Wesley H. Holliday, Bob M. Jacobs, Nathan Lambert, Milan Mossé, Eric Pacuit, Stuart Russell, Hailey Schoelkopf, Emanuel Tewolde, William S. Zwicker

专题命中偏好对齐：alignment(title)；safety(abstract)；分类 cs.CL、cs.AI、cs.CY

Comments 15 pages, 4 figures

2405.20335 2024-05-31 cs.CL 83%

Xwin-LM: Strong and Scalable Alignment Practice for LLMs

Bolin Ni, JingCheng Hu, Yixuan Wei, Houwen Peng, Zheng Zhang, Gaofeng Meng, Han Hu

专题命中偏好对齐：alignment(title,abstract)；DPO(abstract)；分类 cs.CL

2405.18718 2024-05-30 cs.CL 83%

Efficient Model-agnostic Alignment via Bayesian Persuasion

Fengshuo Bai, Mingzhi Wang, Zhaowei Zhang, Boyuan Chen, Yinda Xu, Ying Wen, Yaodong Yang

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

2311.04072 2024-04-16 cs.CL 83%

Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment

Geyang Guo, Ranchi Zhao, Tianyi Tang, Wayne Xin Zhao, Ji-Rong Wen

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

2404.00934 2024-04-04 cs.CL 83%

ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback

Zhenyu Hou, Yilin Niu, Zhengxiao Du, Xiaohan Zhang, Xiao Liu, Aohan Zeng, Qinkai Zheng, Minlie Huang, Hongning Wang, Jie Tang, Yuxiao Dong

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL

2403.16649 2024-03-27 cs.AI 83%

CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment

Feiteng Fang, Liang Zhu, Min Yang, Xi Feng, Jinchang Hou, Qixuan Zhao, Chengming Li, Xiping Hu, Ruifeng Xu

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.AI

2311.00168 2024-02-05 cs.LG 83%

The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback

Nathan Lambert, Roberto Calandra

专题命中偏好对齐：alignment(title)；RLHF(abstract)；safety(abstract)；分类 cs.LG

Comments 11 pages, 5 figures

2312.15241 2023-12-27 cs.AI cs.IR 83%

Measuring Value Alignment

Fazl Barez, Philip Torr

专题命中偏好对齐：alignment(title,abstract)；safety(abstract)；分类 cs.AI

Comments arXiv admin note: text overlap with arXiv:2110.09240 by other authors

Journal ref NeurIPS 2023 MP2 Workshop

2312.07551 2023-12-14 cs.CL 83%

Language Model Alignment with Elastic Reset

Michael Noukhovitch, Samuel Lavoie, Florian Strub, Aaron Courville

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments Published at NeurIPS 2023

2310.11564 2023-10-19 cs.CL 83%

Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging

Joel Jang, Seungone Kim, Bill Yuchen Lin, Yizhong Wang, Jack Hessel, Luke Zettlemoyer, Hannaneh Hajishirzi, Yejin Choi, Prithviraj Ammanabrolu

专题命中偏好对齐：alignment(title,abstract)；RLHF(abstract)；分类 cs.CL

Comments Preprint

2309.14525 2023-09-27 cs.CV cs.CL 83%

Aligning Large Multimodal Models with Factually Augmented RLHF

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell

专题命中偏好对齐：RLHF(title,abstract)；alignment(abstract)；分类 cs.CL

Comments Preprint

2406.18629 2024-06-28 cs.LG cs.AI cs.CL 83%

Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs

Xin Lai, Zhuotao Tian, Yukang Chen, Senqiao Yang, Xiangru Peng, Jiaya Jia

专题命中偏好对齐：DPO(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Code, data, and models are available at https://github.com/dvlab-research/Step-DPO

2406.15890 2024-06-25 cs.LG cs.AI cs.CL cs.GT 83%

Language Alignment via Nash-learning and Adaptive feedback

Ari Azarafrooz, Farshid Faal

专题命中偏好对齐：alignment(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at ICML 2024 Workshop on Models of Human Feedback for AI Alignment, Vienna, Austria

2310.05910 2024-04-11 cs.CL cs.AI cs.LG 83%

SALMON: Self-Alignment with Instructable Reward Models

Zhiqing Sun, Yikang Shen, Hongxin Zhang, Qinhong Zhou, Zhenfang Chen, David Cox, Yiming Yang, Chuang Gan

专题命中偏好对齐：alignment(title,comments)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Previous Title: SALMON: Self-Alignment with Principle-Following Reward Models. Accepted to ICLR 2024. Project page: https://github.com/IBM/SALMON

2608.12192 2026-08-13 cs.AI cs.LG 新提交 82%

How to Spend Your Oracle Budget: Practical Guidance for Protein Structure Prediction Models

如何使用你的专家预算：蛋白质结构预测模型的实用指南

Aleksandra Kalisz, Jack Simons, Krisztina Sinkovics, Noam Ghenassia, Shikha Surana, Henry Moss, Paul Duckworth

机构 * InstaDeep Ltd（InstaDeep公司）； University of Oxford（牛津大学）； Lancaster University（兰卡斯特大学）

专题命中偏好对齐：DPO(summary_cn,abstract)；分类 cs.AI、cs.LG

AI总结本研究针对蛋白质结构预测模型的专家预算约束，通过基准测试FK-steering、DPO、Best K-of-N采样及O3方法，明确不同预算下的最优方法并给出实用选择建议。

Comments Proceedings of the ICML 2026 Workshop on Structured Probabilistic Inference & Generative Modeling (SPIGM)

1. 偏好对齐 3229 篇

Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models

PMoL: Parameter Efficient MoE for Preference Mixing of LLM Alignment

Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment

VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment

SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling

HybridFlow: A Flexible and Efficient RLHF Framework

D2PO: Discriminator-Guided DPO with Response Evaluation Models

Improving Conversational Abilities of Quantized Large Language Models via Direct Preference Alignment

Step-Controlled DPO: Leveraging Stepwise Error for Enhanced Mathematical Reasoning

Understanding Alignment in Multimodal LLMs: A Comprehensive Study

Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback

Averaging log-likelihoods in direct alignment

Does Cross-Cultural Alignment Change the Commonsense Morality of Language Models?

ContraSolver: Self-Alignment of Language Models by Resolving Internal Preference Contradictions

Towards Efficient Exact Optimization of Language Model Alignment

Social Choice Should Guide AI Alignment in Dealing with Diverse Human Feedback

Xwin-LM: Strong and Scalable Alignment Practice for LLMs

Efficient Model-agnostic Alignment via Bayesian Persuasion

Beyond Imitation: Leveraging Fine-grained Quality Signals for Alignment

ChatGLM-RLHF: Practices of Aligning Large Language Models with Human Feedback

CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment

The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback

Measuring Value Alignment

Language Model Alignment with Elastic Reset

Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Merging

Aligning Large Multimodal Models with Factually Augmented RLHF

Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMs

Language Alignment via Nash-learning and Adaptive feedback

SALMON: Self-Alignment with Instructable Reward Models

How to Spend Your Oracle Budget: Practical Guidance for Protein Structure Prediction Models