语言大模型 / LLM

GRPO++: Enhancing Dermatological Reasoning under Low Resource Settings

Ismam Nur Swapnil, Aranya Saha, Tanvir Ahmed Khan, Mohammad Ariful Haque

机构 * Department of Electrical and Electronic Engineering, Bangladesh University of Engineering and Technology (BUET)（电气与电子工程系，孟加拉国工程与技术大学） 纠错

专题命中 后训练与偏好优化 ：language model(abstract)；preference optimization(abstract)；分类 cs.CL、cs.LG

Comments Will be submitted at IEEE JBHI

TreeRPO: Tree Relative Policy Optimization

Zhicheng Yang, Zhijiang Guo, Yinya Huang, Xiaodan Liang, Yiwei Wang, Jing Tang

机构 * The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）） ； The Hong Kong University of Science and Technology（香港科技大学） ； ETH AI Center, ETH Zurich（苏黎世联邦理工学院人工智能中心） ； University of California, Merced（加州大学梅德福分校） ； Sun Yat-sen University（中山大学） ； MBZUAI 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments 13pages, 6 figures

Turning Internal Gap into Self-Improvement: Promoting the Generation-Understanding Unification in MLLMs

Yujin Han, Hao Chen, Andi Han, Zhiheng Wang, Xinyu Liu, Yingya Zhang, Shiwei Zhang, Difan Zou

专题命中 后训练与偏好优化 ：post-training(abstract)；SFT(abstract)；分类 cs.CL、cs.AI

Comments 31 pages, 16 figures, 12 tables

NGRPO: Negative-enhanced Group Relative Policy Optimization

Gongrui Nan, Siye Chen, Jing Huang, Mengyu Lu, Dexun Wang, Chunmei Xie, Weiqi Xiong, Xianzhou Zeng, Qixuan Zhou, Yadong Li, Xingzhong Xu

机构 * Ant Group Shanghai, China（蚂蚁集团上海） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Reward-Weighted Sampling: Enhancing Non-Autoregressive Characteristics in Masked Diffusion LLMs

Daehoon Gwak, Minseo Jung, Junwoo Park, Minho Park, ChaeHun Park, Junha Hyung, Jaegul Choo

机构 * KAIST AI（韩国科学技术院人工智能研究所） ； Applied Artificial Intelligence, Sungkyunkwan University（成均馆大学应用人工智能系） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2025 Main Paper (Long)

Best-of-L: Cross-Lingual Reward Modeling for Mathematical Reasoning

Sara Rajaee, Rochelle Choenni, Ekaterina Shutova, Christof Monz

机构 * Language Technology Lab, University of Amsterdam（阿姆斯特丹大学语言技术实验室） ； ILLC, University of Amsterdam（阿姆斯特丹大学语言学研究所） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Reward Hacking Mitigation using Verifiable Composite Rewards

Mirza Farhan Bin Tarek, Rahmatollah Beheshti

机构 * University of Delaware（特拉华大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Accepted at the 16th ACM Conference on Bioinformatics, Computational Biology, and Health Informatics (ACM-BCB 2025)

Offline RLAIF: Piloting VLM Feedback for RL via SFO

Jacob Beck

机构 * Jacob Beck 1 纠错

专题命中 后训练与偏好优化 ：language model(abstract)；RLHF(abstract)；分类 cs.AI、cs.LG

Comments Code is provided at https://github.com/jacooba/OfflineRLAIF

Journal ref Published at The RLC 2025 Workshop on Reinforcement Learning Beyond Rewards: Ingredients for Developing Generalist Agents

SUDER: Self-Improving Unified Large Multimodal Models for Understanding and Generation with Dual Self-Rewards

Jixiang Hong, Yiran Zhang, Guanzhong Wang, Yi Liu, Ji-Rong Wen, Rui Yan

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学人工智能学院） ； School of Computer Science（计算机科学学院） ； Baidu Inc.（百度公司） ； School of Computer Science, Wuhan University（武汉大学计算机学院） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

R-4B: Incentivizing General-Purpose Auto-Thinking Capability in MLLMs via Bi-Mode Annealing and Reinforce Learning

Qi Yang, Bolin Ni, Shiming Xiang, Han Hu, Houwen Peng, Jie Jiang

机构 * Tencent Hunyuan Team（腾讯文言团队） ； Institute of Automation, CAS（中国科学院自动化研究所） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments 20 pages, 14 figures, 5 tables

RLMR: Reinforcement Learning with Mixed Rewards for Creative Writing

Jianxing Liao, Tian Zhang, Xiao Feng, Yusong Zhang, Rui Yang, Haorui Wang, Bosi Wen, Ziying Wang, Runzhi Shi

机构 * Tencent（腾讯） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

HAEPO: History-Aggregated Exploratory Policy Optimization

Gaurish Trivedi, Alakh Sharma, Kartikey Singh Bhandari, Dhruv Kumar, Pratik Narang, Jagat Sesh Challa

机构 * Birla Institute of Technology and Science, Pilani（比拉理工学院和科学学院，比兰） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Under review

In2x at WMT25 Translation Task

Lei Pang, Hanyi Mao, Quanjia Xiao, HaiXiao Liu, Xiangyi Li

机构 * University of Chicago（芝加哥大学） ； Peking University（北京大学） ； Duxiaoman（杜西亚曼） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

PersRM-R1: Enhance Personalized Reward Modeling with Reinforcement Learning

Mengdi Li, Guanqiao Chen, Xufeng Zhao, Haochen Wen, Shu Yang, Di Wang

专题命中 后训练与偏好优化 ：LLM(abstract)；post-training(abstract)；分类 cs.AI、cs.LG

SafeWork-R1: Coevolving Safety and Intelligence under the AI-45$^{\circ}$ Law

Shanghai AI Lab, :, Yicheng Bao, Guanxu Chen, Mingkang Chen, Yunhao Chen, Chiyu Chen, Lingjie Chen, Sirui Chen, Xinquan Chen, Jie Cheng, Yu Cheng, Dengke Deng, Yizhuo Ding, Dan Ding, Xiaoshan Ding, Yi Ding, Zhichen Dong, Lingxiao Du, Yuyu Fan, Xinshun Feng, Yanwei Fu, Yuxuan Gao, Ruijun Ge, Tianle Gu, Lujun Gui, Jiaxuan Guo, Qianxi He, Yuenan Hou, Xuhao Hu, Hong Huang, Kaichen Huang, Shiyang Huang, Yuxian Jiang, Shanzhe Lei, Jie Li, Lijun Li, Hao Li, Juncheng Li, Xiangtian Li, Yafu Li, Lingyu Li, Xueyan Li, Haotian Liang, Dongrui Liu, Qihua Liu, Zhixuan Liu, Bangwei Liu, Huacan Liu, Yuexiao Liu, Zongkai Liu, Chaochao Lu, Yudong Lu, Xiaoya Lu, Zhenghao Lu, Qitan Lv, Caoyuan Ma, Jiachen Ma, Xiaoya Ma, Zhongtian Ma, Lingyu Meng, Ziqi Miao, Yazhe Niu, Yuezhang Peng, Yuan Pu, Han Qi, Chen Qian, Xingge Qiao, Jingjing Qu, Jiashu Qu, Wanying Qu, Wenwen Qu, Xiaoye Qu, Qihan Ren, Qingnan Ren, Qingyu Ren, Jing Shao, Wenqi Shao, Shuai Shao, Dongxing Shi, Xin Song, Xinhao Song, Yan Teng, Xuan Tong, Yingchun Wang, Xuhong Wang, Shujie Wang, Xin Wang, Yige Wang, Yixu Wang, Yuanfu Wang, Futing Wang, Ruofan Wang, Wenjie Wang, Yajie Wang, Muhao Wei, Xiaoyu Wen, Fenghua Weng, Yuqi Wu, Yingtong Xiong, Xingcheng Xu, Chao Yang, Yue Yang, Yang Yao, Yulei Ye, Zhenyun Yin, Yi Yu, Bo Zhang, Qiaosheng Zhang, Jinxuan Zhang, Yexin Zhang, Yinqiang Zheng, Hefeng Zhou, Zhanhui Zhou, Pengyu Zhu, Qingzi Zhu, Yubo Zhu, Bowen Zhou

机构 * Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） 纠错

专题命中 后训练与偏好优化 ：post-training(abstract)；RLHF(abstract)；分类 cs.CL、cs.AI

Comments 47 pages, 18 figures, authors are listed in alphabetical order by their last names; v3 modifies minor issues

Multimodal LLMs as Customized Reward Models for Text-to-Image Generation

Shijie Zhou, Ruiyi Zhang, Huaisheng Zhu, Branislav Kveton, Yufan Zhou, Jiuxiang Gu, Jian Chen, Changyou Chen

机构 * University at Buffalo（布法罗大学） ； Adobe Research（Adobe研究） ； Pennsylvania State University（宾夕法尼亚州立大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted at ICCV 2025. Code available at https://github.com/sjz5202/LLaVA-Reward

Preference learning made easy: Everything should be understood through win rate

Lily H. Zhang, Rajesh Ranganath

机构 * Center for Data Science, New York University, New York, USA（数据科学中心，纽约大学，纽约，美国） ； Courant Institute, New York University, New York, USA（柯朗研究所，纽约大学，纽约，美国） 纠错

专题命中 后训练与偏好优化 ：SFT(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments ICML 2025

PIPA: Preference Alignment as Prior-Informed Statistical Estimation

Junbo Li, Zhangyang Wang, Qiang Liu

机构 * The University of Texas at Austin, US（德克萨斯大学奥斯汀分校） 纠错

专题命中 后训练与偏好优化 ：language model(abstract)；preference optimization(abstract)；分类 cs.AI、cs.LG

Why Does New Knowledge Create Messy Ripple Effects in LLMs?

Jiaxin Qin, Zixuan Zhang, Manling Li, Pengfei Yu, Heng Ji

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； Stanford University（斯坦福大学） ； Boson AI（博森AI） 纠错

专题命中 后训练与偏好优化 ：language model(abstract)；post-training(abstract)；分类 cs.CL、cs.AI

ViSP: A PPO-Driven Framework for Sarcasm Generation with Contrastive Learning

Changli Wang, Rui Wu, Fang Yin

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Towards Cost-Effective Reward Guided Text Generation

Ahmad Rashid, Ruotian Wu, Rongqi Fan, Hongliang Li, Agustinus Kristiadi, Pascal Poupart

机构 * University of Waterloo（滑铁卢大学） ； Vector Institute（向量研究所） ； Huawei Technologies（华为技术） 纠错

专题命中 后训练与偏好优化 ：language model(abstract)；RLHF(abstract)；分类 cs.CL、cs.LG

Comments 18 pages. Work accepted at ICML 2025

Rethinking Diverse Human Preference Learning through Principal Component Analysis

Feng Luo, Rui Yang, Hao Sun, Chunyuan Deng, Jiarui Yao, Jingyan Shen, Huan Zhang, Hanjie Chen

机构 * Rice University（里士大学） ； University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； University of Cambridge（剑桥大学） ； Columbia University（哥伦比亚大学） 纠错

专题命中 后训练与偏好优化 ：LLM(abstract)；foundation model(abstract)；分类 cs.CL、cs.AI

Comments 14 pages

GFRIEND: Generative Few-shot Reward Inference through EfficieNt DPO

Yiyang Zhao, Huiyu Bai, Xuejiao Zhao

机构 * College of Computing and Data Science,Nanyang Technological University (NTU), Singapore（南洋理工大学计算与数据科学学院） ； Joint NTU-UBC Research Centre of Excellence in Active Living for the Elderly(LILY),NTU,Singapore（南洋理工大学与UBC联合卓越老年活跃生活研究中心（LILY）） 纠错

专题命中 后训练与偏好优化 ：RLHF(abstract)；preference optimization(abstract)；分类 cs.AI、cs.LG

On-Policy RL with Optimal Reward Baseline

Yaru Hao, Li Dong, Xun Wu, Shaohan Huang, Zewen Chi, Furu Wei

机构 * Microsoft Research（微软研究院） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.LG

M3HF: Multi-agent Reinforcement Learning from Multi-phase Human Feedback of Mixed Quality

Ziyan Wang, Zhicheng Zhang, Fei Fang, Yali Du

机构 * Department of Informatics, King’s College London, London, United Kingdom（伦敦大学国王学院信息学系） ； Societal Systems Department, Carnegie Mellon University, Pittsburgh, PA, USA（卡内基梅隆大学社会系统系） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Comments Accepted to ICML 2025

BNPO: Beta Normalization Policy Optimization

Changyi Xiao, Mengdi Zhang, Yixin Cao

机构 * School of Computer Science, Fudan University（复旦大学计算机学院） ； Meituan Group（美团集团） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.AI、cs.LG

Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision

Tej Deep Pala, Panshul Sharma, Amir Zadeh, Chuan Li, Soujanya Poria

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments https://github.com/declare-lab/PathFinder-PRM

Cheems: A Practical Guidance for Building and Evaluating Chinese Reward Models from Scratch

Xueru Wen, Jie Lou, Zichao Li, Yaojie Lu, Xing Yu, Yuqiu Ji, Guohai Xu, Hongyu Lin, Ben He, Xianpei Han, Le Sun, Debing Zhang

机构 * Chinese Information Processing Laboratory, Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所信息处理实验室） ； University of Chinese Academy of Sciences（中国科学院大学） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Accepted to ACL 2025

Teaching with Lies: Curriculum DPO on Synthetic Negatives for Hallucination Detection

Shrey Pandit, Ashwin Vinod, Liu Leqi, Ying Ding

机构 * The University of Texas at Austin（德克萨斯大学奥斯汀分校） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI

Comments Code and dataset are available at https://teachingwithlies.github.io/

From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignment

Jia-Nan Li, Jian Guan, Songhao Wu, Wei Wu, Rui Yan

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI