语言大模型 / LLM - arXivDaily 专题

2505.18126 2025-09-30 cs.LG cs.AI cs.CL 87%

Reward Model Overoptimisation in Iterated RLHF

Lorenz Wolf, Robert Kirk, Mirco Musolesi

机构 * UCL Centre for Artificial Intelligence（伦敦大学学院人工智能中心）； Department of Computer Science（计算机科学系）； University College London（伦敦大学学院）； UK AI Security Institute（英国人工智能安全研究所）； Department of Computer Science and Engineering（计算机科学与工程系）； University of Bologna（博洛尼亚大学）

专题命中后训练与偏好优化：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 24 pages, 17 figures, 6 tables

2509.16990 2025-09-23 cs.CL cs.AI cs.LG cs.SD eess.AS 87%

Advancing Speech Understanding in Speech-Aware Language Models with GRPO

Avishai Elmakies, Hagai Aronowitz, Nimrod Shabtay, Eli Schwartz, Ron Hoory, Avihu Dekel

机构 * IBM Research（IBM研究院）

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；SFT(abstract)；分类 cs.CL、cs.AI、cs.LG

2504.05946 2025-09-08 eess.SY cs.SY 87%

InstructMPC: A Human-LLM-in-the-Loop Framework for Context-Aware Control

Ruixiang Wu, Jiahao Ai, Tongxin Li

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

2508.19918 2025-09-01 cs.IR 87%

Refining Text Generation for Realistic Conversational Recommendation via Direct Preference Optimization

Manato Tajiri, Michimasa Inaba

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Accepted to EMNLP 2025 Main Conference

2508.07768 2025-08-12 cs.LG cs.AI cs.CL 87%

Pareto Multi-Objective Alignment for Language Models

Qiang He, Setareh Maghsudi

机构 * Ruhr University Bochum（鲁尔大学波恩）

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at ECML/PKDD 2025

2507.20227 2025-08-05 cs.IR 87%

CTR-Driven Ad Text Generation via Online Feedback Preference Optimization

Yanda Chen, Zihui Ren, Qixiang Gao, Jiale Chen, Si Chen, Xubin Li, Tiezheng Ge, Bo Zheng

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments 13 pages, 7 figures, 8 tables

2412.04628 2025-06-23 cs.LG cs.AI cs.CL 87%

Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts

Taneesh Gupta, Rahul Madhavan, Xuchao Zhang, Nagarajan Natarajan, Chetan Bansal, Saravan Rajmohan

专题命中后训练与偏好优化：preference optimization(title,abstract)；language model(abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

2506.14574 2025-06-18 cs.LG cs.AI cs.CL 87%

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization

Mingkang Zhu, Xi Chen, Zhongdao Wang, Bei Yu, Hengshuang Zhao, Jiaya Jia

机构 * The Chinese University of Hong Kong（香港中文大学）； The University of Hong Kong（香港大学）； The Hong Kong University of Science（香港科学大学）； Huawei（华为）

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICML 2025

2506.08712 2025-06-13 cs.CL cs.AI cs.LG 87%

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

Hee Suk Yoon, Eunseop Yoon, Mark Hasegawa-Johnson, Sungwoong Kim, Chang D. Yoo

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）； University of Illinois Urbana-Champaign (UIUC)（伊利诺伊大学厄巴纳-香槟分校）； Korea University（韩国大学）

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments ICML 2025

2502.18293 2025-06-10 cs.LG cs.AI cs.CL 87%

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection

Taneesh Gupta, Rahul Madhavan, Xuchao Zhang, Chetan Bansal, Saravan Rajmohan

机构 * Microsoft（微软）

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at ICML 2025

2506.05260 2025-06-06 cs.CV 87%

LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs

Xiaodong Wang, Jinfa Huang, Li Yuan, Peixi Peng

机构 * Peking University（北京大学）； Peng Cheng Laboratory（鹏城实验室）

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Code: https://github.com/Wang-Xiaodong1899/LeanPO

2505.18720 2025-05-27 cs.CL cs.AI cs.LG 87%

Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization

Meng Li, Guangda Huzhang, Haibo Zhang, Xiting Wang, Anxiang Zeng

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院）； LLM Team, Shopee Pte. Ltd.（Shopee Pte. Ltd. 语言模型团队）； Beijing Key Laboratory of Research on Large Models and Intelligent Governance and Engineering Research Center of Next-Generation Intelligent Search and Recommendation, MOE（北京大型模型与智能治理研究重点实验室）

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 24 pages, 11 figures. Accepted by ACL 2025 (main)

2505.10597 2025-05-20 cs.LG cs.AI cs.CL 87%

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

Jiazheng Zhang, Wenqing Jing, Zizhuo Zhang, Zhiheng Xi, Shihan Dou, Rongxiang Weng, Jiahuan Li, Jingang Wang, Mingxu Chai, Shibo Hong, Tao Gui, Qi Zhang

机构 * NLP Group, Fudan University（复旦大学自然语言处理组）； Hong Kong Baptist University（香港 Baptist 大学）； LLM Team, Meituan Inc（美团公司大语言模型团队）

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2410.04612 2025-04-25 cs.LG cs.AI cs.CL 87%

Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF

Zhaolin Gao, Wenhao Zhan, Jonathan D. Chang, Gokul Swamy, Kianté Brantley, Jason D. Lee, Wen Sun

机构 * Cornell University（康奈尔大学）； Princeton University（普林斯顿大学）； Databricks Mosaic Research（Databricks 混合研究）； Carnegie Mellon University（卡内基梅隆大学）； Harvard University（哈佛大学）

专题命中后训练与偏好优化：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2410.13828 2025-04-23 cs.LG cs.AI cs.CL 87%

A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement

Hui Yuan, Yifan Zeng, Yue Wu, Huazheng Wang, Mengdi Wang, Liu Leqi

机构 * The University of Texas at Austin（德克萨斯大学奥斯汀分校）； Princeton Language & Intelligence（普林斯顿语言与智能）

专题命中后训练与偏好优化：language model(title,abstract)；RLHF(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

2504.15619 2025-04-23 cs.CV 87%

AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization

Jinda Lu, Jinghan Li, Yuan Gao, Junkang Wu, Jiancan Wu, Xiang Wang, Xiangnan He

机构 * University of Science and Technology of China（中国科学技术大学）

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；foundation model(abstract)

2503.10723 2025-03-17 cs.CL cs.AI cs.IR cs.LG 87%

RankPO: Preference Optimization for Job-Talent Matching

Yafei Zhang, Murray Wang, Yu Wang, Xiaohui Wang

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 15 pages, 3 figures, 7 tables

2503.06781 2025-03-11 cs.CL cs.AI cs.LG 87%

Dr Genre: Reinforcement Learning from Decoupled LLM Feedback for Generic Text Rewriting

Yufei Li, John Nham, Ganesh Jawahar, Lei Shu, David Uthus, Yun-Hsuan Sung, Chengrun Yang, Itai Rolnick, Yi Qiao, Cong Liu

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 29 pages, 4 figures, 25 tables

2407.13399 2025-02-19 cs.AI cs.CL cs.LG 87%

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization

Audrey Huang, Wenhao Zhan, Tengyang Xie, Jason D. Lee, Wen Sun, Akshay Krishnamurthy, Dylan J. Foster

专题命中后训练与偏好优化：preference optimization(title,abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.12206 2025-02-19 cs.AI cs.CL cs.LG 87%

Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?

Yufei He, Yuexin Li, Jiaying Wu, Yuan Sui, Yulin Chen, Bryan Hooi

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.04371 2025-02-10 cs.AI cs.CL cs.LG 87%

PerPO: Perceptual Preference Optimization via Discriminative Rewarding

Zining Zhu, Liang Zhao, Kangheng Lin, Jinze Yang, En Yu, Chenglong Liu, Haoran Wei, Jianjian Sun, Zheng Ge, Xiangyu Zhang

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2501.13120 2025-01-24 cs.CL cs.AI cs.LG cs.MA 87%

Multilinguality in LLM-Designed Reward Functions for Restless Bandits: Effects on Task Performance and Fairness

Ambreesh Parthasarathy, Chandrasekar Subramanian, Ganesh Senrayan, Shreyash Adappanavar, Aparna Taneja, Balaraman Ravindran, Milind Tambe

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at the AAAI-2025 Deployable AI Workshop

2501.03271 2025-01-22 cs.LG cs.AI cs.CL 87%

DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization

Amitava Das, Suranjana Trivedy, Danush Khanna, Rajarshi Roy, Gurpreet Singh, Basab Ghosh, Yaswanth Narsupalli, Vinija Jain, Vasu Sharma, Aishwarya Naresh Reganti, Aman Chadha

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.01766 2024-12-20 cs.CL cs.AI cs.CY cs.LG econ.GN q-fin.EC 87%

LLM Voting: Human Choices and AI Collective Decision Making

Joshua C. Yang, Damian Dailisan, Marcin Korecki, Carina I. Hausladen, Dirk Helbing

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted in AAAI Conference on AI, Ethics, and Society (AIES)

2404.13846 2024-12-04 cs.LG cs.AI cs.CL 87%

Filtered Direct Preference Optimization

Tetsuro Morimura, Mitsuki Sakamoto, Yuu Jinnai, Kenshi Abe, Kaito Ariu

专题命中后训练与偏好优化：preference optimization(title,abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments EMNLP 2024

2410.22891 2024-10-31 cs.LG cs.AI cs.CL 87%

VPO: Leveraging the Number of Votes in Preference Optimization

Jae Hyeon Cho, Minkyung Park, Byung-Jun Lee

专题命中后训练与偏好优化：preference optimization(title,abstract)；language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2410.14872 2024-10-24 cs.LG cs.AI cs.CL 87%

How to Evaluate Reward Models for RLHF

Evan Frick, Tianle Li, Connor Chen, Wei-Lin Chiang, Anastasios N. Angelopoulos, Jiantao Jiao, Banghua Zhu, Joseph E. Gonzalez, Ion Stoica

专题命中后训练与偏好优化：RLHF(title,abstract)；LLM(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2404.11049 2024-10-22 cs.LG cs.AI cs.CL 87%

Stepwise Alignment for Constrained Language Model Policy Optimization

Akifumi Wachi, Thien Q. Tran, Rei Sato, Takumi Tanabe, Youhei Akimoto

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at NeurIPS 2024. Code and models are available at https://github.com/line/sacpo

2402.03469 2024-10-08 cs.LG cs.AI cs.CL 87%

Rethinking the Role of Proxy Rewards in Language Model Alignment

Sungdong Kim, Minjoon Seo

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to EMNLP 2024 main conference

2409.17545 2024-09-30 cs.CL cs.AI cs.LG 87%

Modulated Intervention Preference Optimization (MIPO): Keep the Easy, Refine the Difficult

Cheolhun Jang

专题命中后训练与偏好优化：preference optimization(title,abstract)；SFT(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 8pages, submitted to AAAI 2025

1. 后训练与偏好优化 4516 篇

Reward Model Overoptimisation in Iterated RLHF

Advancing Speech Understanding in Speech-Aware Language Models with GRPO

InstructMPC: A Human-LLM-in-the-Loop Framework for Context-Aware Control

Refining Text Generation for Realistic Conversational Recommendation via Direct Preference Optimization

Pareto Multi-Objective Alignment for Language Models

CTR-Driven Ad Text Generation via Online Feedback Preference Optimization

Multi-Preference Optimization: Generalizing DPO via Set-Level Contrasts

TGDPO: Harnessing Token-Level Reward Guidance for Enhancing Direct Preference Optimization

ConfPO: Exploiting Policy Model Confidence for Critical Token Selection in Preference Optimization

AMPO: Active Multi-Preference Optimization for Self-play Preference Selection

LeanPO: Lean Preference Optimization for Likelihood Alignment in Video-LLMs

Optimal Transport-Based Token Weighting scheme for Enhanced Preference Optimization

Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment

Regressing the Relative Future: Efficient Policy Optimization for Multi-turn RLHF

A Common Pitfall of Margin-based Language Model Alignment: Gradient Entanglement

AdaViP: Aligning Multi-modal LLMs via Adaptive Vision-enhanced Preference Optimization

RankPO: Preference Optimization for Job-Talent Matching

Dr Genre: Reinforcement Learning from Decoupled LLM Feedback for Generic Text Rewriting

Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization

Evaluating the Paperclip Maximizer: Are RL-Based Language Models More Likely to Pursue Instrumental Goals?

PerPO: Perceptual Preference Optimization via Discriminative Rewarding

Multilinguality in LLM-Designed Reward Functions for Restless Bandits: Effects on Task Performance and Fairness

DPO Kernels: A Semantically-Aware, Kernel-Enhanced, and Divergence-Rich Paradigm for Direct Preference Optimization

LLM Voting: Human Choices and AI Collective Decision Making

Filtered Direct Preference Optimization

VPO: Leveraging the Number of Votes in Preference Optimization

How to Evaluate Reward Models for RLHF

Stepwise Alignment for Constrained Language Model Policy Optimization

Rethinking the Role of Proxy Rewards in Language Model Alignment

Modulated Intervention Preference Optimization (MIPO): Keep the Easy, Refine the Difficult