语言大模型 / LLM - arXivDaily 专题

2406.03199 2025-03-13 cs.CL cs.AI cs.LG 80%

Bayesian WeakS-to-Strong from Text Classification to Generation

Ziyun Cui, Ziyang Zhang, Guangzhi Sun, Wen Wu, Chao Zhang

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by ICLR2025

2502.18411 2025-03-04 cs.CV 80%

OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference

Xiangyu Zhao, Shengyuan Ding, Zicheng Zhang, Haian Huang, Maosong Cao, Weiyun Wang, Jiaqi Wang, Xinyu Fang, Wenhai Wang, Guangtao Zhai, Haodong Duan, Hua Yang, Kai Chen

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；SFT(abstract)；preference optimization(abstract)

2502.20356 2025-02-28 cs.CL cs.AI cs.LG 80%

Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs

Kuan Lok Zhou, Jiayi Chen, Siddharth Suresh, Reuben Narad, Timothy T. Rogers, Lalit K Jain, Robert D Nowak, Bob Mankoff, Jifan Zhang

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2409.04421 2025-01-20 cs.CL cs.AI cs.LG 80%

RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs

Jiaxing Wu, Lin Ning, Luyang Liu, Harrison Lee, Neo Wu, Chao Wang, Sushant Prakash, Shawn O'Banion, Bradley Green, Jun Xie

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments AAAI 2025

2310.05857 2025-01-07 cs.CL cs.AI cs.LG 80%

Improving Summarization with Human Edits

Zonghai Yao, Benjamin J Schloss, Sai P. Selvaraj

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Proceedings of the Main Conference on Empirical Methods in Natural Language Processing (EMNLP) 2023

2406.10522 2024-12-19 cs.LG cs.AI cs.CL 80%

Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning

Jifan Zhang, Lalit Jain, Yang Guo, Jiayi Chen, Kuan Lok Zhou, Siddharth Suresh, Andrew Wagenmaker, Scott Sievert, Timothy Rogers, Kevin Jamieson, Robert Mankoff, Robert Nowak

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2404.12535 2024-12-17 cs.LG cs.AI cs.CL 80%

Is There No Such Thing as a Bad Question? H4R: HalluciBot For Ratiocination, Rewriting, Ranking, and Routing

William Watson, Nicole Cho, Nishan Srishankar

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at The 39th Annual AAAI Conference on Artificial Intelligence (AAAI 2025)

2406.16235 2024-11-11 cs.CL cs.AI cs.CR cs.LG 80%

Preference Tuning For Toxicity Mitigation Generalizes Across Languages

Xiaochen Li, Zheng-Xin Yong, Stephen H. Bach

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Findings of EMNLP 2024

2401.16635 2024-10-23 cs.LG cs.AI cs.CL 80%

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

Shun Zhang, Zhenfang Chen, Sunli Chen, Yikang Shen, Zhiqing Sun, Chuang Gan

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2407.17546 2024-07-26 cs.LG cs.AI cs.CL 80%

Exploring Domain Robust Lightweight Reward Models based on Router Mechanism

Hyuk Namgoong, Jeesu Jung, Sangkeun Jung, Yoonhyung Roh

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；small language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments This paper is accepted for ACL 2024

2406.01462 2024-07-17 cs.LG cs.AI cs.CL 80%

The Importance of Online Data: Understanding Preference Fine-tuning via Coverage

Yuda Song, Gokul Swamy, Aarti Singh, J. Andrew Bagnell, Wen Sun

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.13228 2024-07-04 cs.CL cs.AI cs.LG 80%

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

Arka Pal, Deep Karkhanis, Samuel Dooley, Manley Roberts, Siddartha Naidu, Colin White

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.03746 2024-06-18 cs.CV 80%

Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback

Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, Jonghyun Choi

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；SFT(abstract)

Comments ACL 2024

2406.08673 2024-06-14 cs.CL cs.AI cs.LG 80%

HelpSteer2: Open-source dataset for training top-performing reward models

Zhilin Wang, Yi Dong, Olivier Delalleau, Jiaqi Zeng, Gerald Shen, Daniel Egert, Jimmy J. Zhang, Makesh Narsimhan Sreedhar, Oleksii Kuchaiev

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.02479 2024-06-11 cs.LG cs.AI cs.CL cs.HC 80%

BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback

Gaurav Pandey, Yatin Nandwani, Tahira Naseem, Mayank Mishra, Guangxuan Xu, Dinesh Raghu, Sachindra Joshi, Asim Munawar, Ramón Fernandez Astudillo

专题命中后训练与偏好优化：language model(abstract)；RLHF(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at ICML 2024 (main conference)

2405.11165 2024-05-30 cs.CV 80%

Automated Multi-level Preference for MLLMs

Mengxi Zhang, Wenhao Wu, Yu Lu, Yuxin Song, Kang Rong, Huanjin Yao, Jianbo Zhao, Fanglong Liu, Yifan Sun, Haocheng Feng, Jingdong Wang

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；preference optimization(abstract)

Comments Preprint

2311.14455 2024-04-30 cs.AI cs.CL cs.CR cs.LG 80%

Universal Jailbreak Backdoors from Poisoned Human Feedback

Javier Rando, Florian Tramèr

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted as conference paper in ICLR 2024

2404.07491 2024-04-12 cs.SE 80%

Neural Fault Injection: Generating Software Faults from Natural Language

Domenico Cotroneo, Pietro Liguori

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments 2024 54th Annual IEEE/IFIP International Conference on Dependable Systems and Networks - Disrupt

2309.10691 2024-03-13 cs.CL cs.AI cs.LG 80%

MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback

Xingyao Wang, Zihan Wang, Jiateng Liu, Yangyi Chen, Lifan Yuan, Hao Peng, Heng Ji

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2024. Code is available on our project website: https://xingyaoww.github.io/mint-bench

2402.01694 2024-02-06 cs.CL cs.AI cs.LG 80%

ARGS: Alignment as Reward-Guided Search

Maxim Khanov, Jirayu Burapacheep, Yixuan Li

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICLR 2024

2311.14115 2024-01-11 cs.LG cs.AI cs.CL 80%

A density estimation perspective on learning from pairwise human preferences

Vincent Dumoulin, Daniel D. Johnson, Pablo Samuel Castro, Hugo Larochelle, Yann Dauphin

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2305.14387 2024-01-09 cs.LG cs.AI cs.CL 80%

AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback

Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto

专题命中后训练与偏好优化：LLM(abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Spotlight at NeurIPS 2023

2310.16048 2023-10-25 cs.AI cs.CL cs.CY cs.HC cs.LG 80%

AI Alignment and Social Choice: Fundamental Limitations and Policy Implications

Abhilash Mishra

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 10 pages, no figures

2307.15217 2023-09-12 cs.AI cs.CL cs.LG 80%

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Stephen Casper, Xander Davies, Claudia Shi, Thomas Krendl Gilbert, Jérémy Scheurer, Javier Rando, Rachel Freedman, Tomasz Korbak, David Lindner, Pedro Freire, Tony Wang, Samuel Marks, Charbel-Raphaël Segerie, Micah Carroll, Andi Peng, Phillip Christoffersen, Mehul Damani, Stewart Slocum, Usman Anwar, Anand Siththaranjan, Max Nadeau, Eric J. Michaud, Jacob Pfau, Dmitrii Krasheninnikov, Xin Chen, Lauro Langosco, Peter Hase, Erdem Bıyık, Anca Dragan, David Krueger, Dorsa Sadigh, Dylan Hadfield-Menell

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2305.12219 2023-05-26 cs.LG cs.AI cs.CL 80%

Collaborative Development of NLP models

Fereshte Khani, Marco Tulio Ribeiro

专题命中后训练与偏好优化：large language model(abstract)；language model(abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

2507.09973 2025-07-15 cs.CL cs.AI 80%

Tiny Reward Models

Sarah Pan

机构 * Massachusetts Institute of Technology（麻省理工学院）

专题命中后训练与偏好优化：language model(abstract)；RLHF(abstract)；prompting(abstract)；分类 cs.CL、cs.AI

Comments 2025 ICML Efficient Systems for Foundation Models Workshop

2608.20011 2026-08-21 cs.AI cs.CV 新提交 79%

Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hacking

流偏好优化中的流形漂移：奖励黑客的根本原因

Yansen Han, Shengyi Liao, Yuanxing Zhang, Pengfei Wan, Tao Lin

机构 * Zhejiang University（浙江大学）； Kuaishou Technology（快手科技）； Westlake University（西湖大学）

专题命中后训练与偏好优化：preference optimization(title,abstract)；分类 cs.AI

AI总结本文针对流偏好优化中流形漂移导致奖励黑客的问题，提出ThermoDPO及其加权变体，在玩具基准和SD3.5-M数据集上均取得优于FlowDPO等方法的性能。

2608.19182 2026-08-20 cs.RO cs.AI 新提交 79%

ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning

ADEPT：通过强化学习利用预训练和后训练提升灵巧性

Jayjun Lee, Jessica Yin, Asif Rana, Nicholas Blauch, Sam Mady, Mohak Bhardwaj, Nima Fazeli, Nathan Ratliff, Karl Van Wyk, Ankur Handa

机构 * NVIDIA Corporation（英伟达公司）； University of Michigan（密歇根大学）

专题命中后训练与偏好优化：post-training(title,abstract)；分类 cs.AI

AI总结该研究提出ADEPT强化学习框架，通过预训练和后训练实现高自由度机器人的可仿真到现实迁移的灵巧性，在两款机器人本体上完成长程任务，达到人类水平灵巧速度。

Comments Project page: https://adept-dexterity.github.io/

2608.11226 2026-08-13 cs.AI cs.SY eess.SY 新提交 79%

Cutting AI Datacenter Energy with Reinforcement Learning: Measured Power Control of LLM Training from One GPU to the Fleet

用强化学习降低AI数据中心能耗：从单GPU到集群的大语言模型训练实测功率控制

Eliseo Curcio

专题命中后训练与偏好优化：LLM(title)；post-training(abstract)；分类 cs.AI

AI总结该研究针对大语言模型训练的GPU功耗问题，提出PPO元控制器调整生成参数，在单GPU及集群环境下降低功率违规、提升输出与能效，验证了超额订阅的可行性。

2608.09568 2026-08-11 cs.CL 新提交 79%

Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization

Se-DPO：用于直接偏好优化的自进化令牌信用

Wenxiao Zhao, Shu Wang, Ying Nian Wu

专题命中后训练与偏好优化：preference optimization(title,abstract)；分类 cs.CL

AI总结 Se-DPO是一种在DPO训练中基于模型自身内部信号动态校准令牌信用的方法，无需外部模型，在AlpacaEval 2和Arena-Hard上较DPO分别提升9.8和12.2个百分点。

Comments 16 pages, 2 figures, COLM2026

1. 后训练与偏好优化 4535 篇

Bayesian WeakS-to-Strong from Text Classification to Generation

OmniAlign-V: Towards Enhanced Alignment of MLLMs with Human Preference

Bridging the Creativity Understanding Gap: Small-Scale Human Alignment Enables Expert-Level Humor Ranking in LLMs

RLPF: Reinforcement Learning from Prediction Feedback for User Summarization with LLMs

Improving Summarization with Human Edits

Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning

Is There No Such Thing as a Bad Question? H4R: HalluciBot For Ratiocination, Rewriting, Ranking, and Routing

Preference Tuning For Toxicity Mitigation Generalizes Across Languages

Improving Reinforcement Learning from Human Feedback with Efficient Reward Model Ensemble

Exploring Domain Robust Lightweight Reward Models based on Router Mechanism

The Importance of Online Data: Understanding Preference Fine-tuning via Coverage

Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positive

Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback

HelpSteer2: Open-source dataset for training top-performing reward models

BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback

Automated Multi-level Preference for MLLMs

Universal Jailbreak Backdoors from Poisoned Human Feedback

Neural Fault Injection: Generating Software Faults from Natural Language

MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback

ARGS: Alignment as Reward-Guided Search

A density estimation perspective on learning from pairwise human preferences

AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback

AI Alignment and Social Choice: Fundamental Limitations and Policy Implications

Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback

Collaborative Development of NLP models

Tiny Reward Models

Manifold Drift in Flow Preference Optimization: A Root Cause of Reward Hacking

ADEPT: Accelerating Dexterity via Pre-Training and Post-Training using Reinforcement Learning

Cutting AI Datacenter Energy with Reinforcement Learning: Measured Power Control of LLM Training from One GPU to the Fleet

Se-DPO: Self-Evolving Token Credit for Direct Preference Optimization