语言大模型 / LLM - arXivDaily 专题

2501.15453 2025-01-29 cs.CL 77%

Data-adaptive Safety Rules for Training Reward Models

Xiaomin Li, Mingye Gao, Zhiwei Zhang, Jingxuan Fan, Weiyu Li

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL

2411.04991 2025-01-28 cs.AI 77%

Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives

Hao Sun, Yunyi Shen, Jean-Francois Ton

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

2406.19185 2025-01-17 cs.LG 77%

Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion

Yannis Flet-Berliac, Nathan Grinsztajn, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Mohammad Gheshlaghi Azar, Olivier Pietquin, Matthieu Geist

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.LG

Comments EMNLP 2024

2412.14308 2025-01-07 cs.SE cs.LG 77%

Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation

Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, Alexey Svyatkovskiy

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

Comments This work was intended as a replacement of arXiv:2310.02368 and any subsequent updates will appear there

2310.02368 2025-01-07 cs.SE cs.LG 77%

Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation

Benjamin Steenhoek, Michele Tufano, Neel Sundaresan, Alexey Svyatkovskiy

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.LG

Comments Accepted to DeepTest 2025 (ICSE Workshop). Previously this version appeared as arXiv:2412.14308 which was submitted as a new work by accident

2501.01054 2025-01-03 cs.CL cs.SE 77%

Dynamic Scaling of Unit Tests for Code Reward Modeling

Zeyao Ma, Xiaokang Zhang, Jing Zhang, Jifan Yu, Sijia Luo, Jie Tang

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments Homepage: https://code-reward-model.github.io/

2410.02504 2025-01-03 stat.ML cs.LG 77%

Dual Active Learning for Reinforcement Learning from Human Feedback

Pangpang Liu, Chengchun Shi, Will Wei Sun

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.LG

2405.14655 2024-12-03 cs.LG 77%

Multi-turn Reinforcement Learning from Preference Human Feedback

Lior Shani, Aviv Rosenberg, Asaf Cassel, Oran Lang, Daniele Calandriello, Avital Zipori, Hila Noga, Orgad Keller, Bilal Piot, Idan Szpektor, Avinatan Hassidim, Yossi Matias, Rémi Munos

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.LG

2412.00074 2024-12-03 cs.CL 77%

Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness

Avinash Amballa, Durga Sandeep Saluru, Gayathri Akkinapalli, Abhishek Sureddy, Akshay Kumar Sureddy

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL

Comments 18 pages

2406.06874 2024-12-03 cs.AI cs.HC cs.RO 77%

Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment

Chenliang Li, Siliang Zeng, Zeyi Liao, Jiaxiang Li, Dongyeop Kang, Alfredo Garcia, Mingyi Hong

专题命中后训练与偏好优化：foundation model(abstract)；SFT(abstract)；RLHF(abstract)；分类 cs.AI

2410.20290 2024-11-04 cs.CL 77%

Fast Best-of-N Decoding via Speculative Rejection

Hanshi Sun, Momin Haider, Ruiqi Zhang, Huitao Yang, Jiahao Qiu, Ming Yin, Mengdi Wang, Peter Bartlett, Andrea Zanette

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；post-training(abstract)；分类 cs.CL

Comments NeurIPS 2024

2405.12739 2024-10-14 cs.LG 77%

SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling

Xingzhou Lou, Junge Zhang, Jian Xie, Lifeng Liu, Dong Yan, Kaiqi Huang

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.LG

2407.01470 2024-10-08 cs.CL 77%

DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging

Tzu-Han Lin, Chen-An Li, Hung-yi Lee, Yun-Nung Chen

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL

Comments In the 2024 Conference on Empirical Methods in Natural Language Processing (EMNLP 2024). The code for our work is available at: https://github.com/MiuLab/DogeRM

2409.06903 2024-09-12 cs.LG 77%

Semi-Supervised Reward Modeling via Iterative Self-Training

Yifei He, Haoxiang Wang, Ziyan Jiang, Alexandros Papangelis, Han Zhao

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.LG

2403.18349 2024-08-09 cs.CL 77%

Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback

Hongshen Xu, Zichen Zhu, Situo Zhang, Da Ma, Shuai Fan, Lu Chen, Kai Yu

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

2407.02477 2024-07-03 cs.CV cs.CL 77%

Understanding Alignment in Multimodal LLMs: A Comprehensive Study

Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, Peter Grasch

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL

2401.11458 2024-07-03 cs.CL 77%

Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback

Songyang Gao, Qiming Ge, Wei Shen, Shihan Dou, Junjie Ye, Xiao Wang, Rui Zheng, Yicheng Zou, Zhi Chen, Hang Yan, Qi Zhang, Dahua Lin

专题命中后训练与偏好优化：LLM(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL

Comments Accepted by ICML2024, I'm still preparing a better vision

2406.19644 2024-07-02 cs.AI 77%

Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs

Zichao Shen, Tianchen Zhu, Qingyun Sun, Shiqi Gao, Jianxin Li

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

Comments accepted by IJCAI 2024 GAAMAL

2406.19188 2024-06-28 cs.LG 77%

Averaging log-likelihoods in direct alignment

Nathan Grinsztajn, Yannis Flet-Berliac, Mohammad Gheshlaghi Azar, Florian Strub, Bill Wu, Eugene Choi, Chris Cremer, Arash Ahmadian, Yash Chandak, Olivier Pietquin, Matthieu Geist

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.LG

2406.18346 2024-06-27 cs.AI 77%

AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations

Adam Dahlgren Lindström, Leila Methnani, Lea Krause, Petter Ericson, Íñigo Martínez de Rituerto de Troya, Dimitri Coelho Mollo, Roel Dobbe

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI

Comments 12 pages, 1 table, to be submitted

2406.16486 2024-06-25 cs.AI 77%

Towards Comprehensive Preference Data Collection for Reward Modeling

Yulan Hu, Qingyang Li, Sheng Ouyang, Ge Chen, Kaihui Chen, Lijun Mei, Xucheng Ye, Fuzheng Zhang, Yong Liu

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI

2406.10991 2024-06-18 cs.CL 77%

Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answers

Tianhua Zhang, Kun Li, Hongyin Luo, Xixin Wu, James Glass, Helen Meng

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL

2406.08713 2024-06-14 cs.AI cs.CV 77%

Batch-Instructed Gradient for Prompt Evolution:Systematic Prompt Optimization for Enhanced Text-to-Image Synthesis

Xinrui Yang, Zhuohan Wang, Anthony Hu

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

2406.08469 2024-06-13 cs.LG 77%

PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences

Daiwei Chen, Yi Chen, Aniket Rege, Ramya Korlakai Vinayak

专题命中后训练与偏好优化：language model(abstract)；foundation model(abstract)；pretraining(abstract)；分类 cs.LG

Comments 22 pages, 14 figures, 5 tables

2402.09764 2024-05-31 cs.AI 77%

Aligning Crowd Feedback via Distributional Preference Reward Modeling

Dexun Li, Cong Zhang, Kuicai Dong, Derrick Goh Xin Deik, Ruiming Tang, Yong Liu

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.AI

2403.01197 2024-04-30 cs.CL 77%

DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling

Shanghaoran Quan

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL

Comments 23 pages, 8 figures

2403.16649 2024-03-27 cs.AI 77%

CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment

Feiteng Fang, Liang Zhu, Min Yang, Xi Feng, Jinchang Hou, Qixuan Zhao, Chengming Li, Xiping Hu, Ruifeng Xu

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.AI

2310.02743 2024-03-12 cs.LG 77%

Reward Model Ensembles Help Mitigate Overoptimization

Thomas Coste, Usman Anwar, Robert Kirk, David Krueger

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.LG

Comments Accepted at ICLR 2024

2402.14740 2024-02-27 cs.LG 77%

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

Arash Ahmadian, Chris Cremer, Matthias Gallé, Marzieh Fadaee, Julia Kreutzer, Olivier Pietquin, Ahmet Üstün, Sara Hooker

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.LG

Comments 27 pages, 7 figures, 2 tables

2402.07896 2024-02-14 cs.CL 77%

Suppressing Pink Elephants with Direct Principle Feedback

Louis Castricato, Nathan Lile, Suraj Anand, Hailey Schoelkopf, Siddharth Verma, Stella Biderman

专题命中后训练与偏好优化：LLM(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL

Comments 8 pages, 6 figures

1. 后训练与偏好优化 4535 篇

Data-adaptive Safety Rules for Training Reward Models

Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives

Contrastive Policy Gradient: Aligning LLMs on sequence-level scores in a supervised-friendly fashion

Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation

Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generation

Dynamic Scaling of Unit Tests for Code Reward Modeling

Dual Active Learning for Reinforcement Learning from Human Feedback

Multi-turn Reinforcement Learning from Preference Human Feedback

Safe to Serve: Aligning Instruction-Tuned Models for Safety and Helpfulness

Learning Reward and Policy Jointly from Demonstration and Preference Improves Alignment

Fast Best-of-N Decoding via Speculative Rejection

SPO: Multi-Dimensional Preference Sequential Alignment With Implicit Reward Modeling

DogeRM: Equipping Reward Models with Domain Knowledge through Model Merging

Semi-Supervised Reward Modeling via Iterative Self-Training

Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback

Understanding Alignment in Multimodal LLMs: A Comprehensive Study

Linear Alignment: A Closed-form Solution for Aligning Human Preferences without Tuning and Feedback

Beyond Human Preferences: Exploring Reinforcement Learning Trajectory Evaluation and Improvement through LLMs

Averaging log-likelihoods in direct alignment

AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations

Towards Comprehensive Preference Data Collection for Reward Modeling

Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answers

Batch-Instructed Gradient for Prompt Evolution:Systematic Prompt Optimization for Enhanced Text-to-Image Synthesis

PAL: Pluralistic Alignment Framework for Learning from Heterogeneous Preferences

Aligning Crowd Feedback via Distributional Preference Reward Modeling

DMoERM: Recipes of Mixture-of-Experts for Effective Reward Modeling

CLHA: A Simple yet Effective Contrastive Learning Framework for Human Alignment

Reward Model Ensembles Help Mitigate Overoptimization

Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs

Suppressing Pink Elephants with Direct Principle Feedback