语言大模型 / LLM

Learning Explainable Dense Reward Shapes via Bayesian Optimization

Ryan Koo, Ian Yang, Vipul Raheja, Mingyi Hong, Kwang-Sung Jun, Dongyeop Kang

机构 * University of Minnesota（明尼苏达大学） ； Georgia Institute of Technology（佐治亚理工学院） ； Grammarly ； University of Arizona（亚利桑那大学） 纠错

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Exploring Data Scaling Trends and Effects in Reinforcement Learning from Human Feedback

Wei Shen, Guanlin Liu, Zheng Wu, Ruofei Zhu, Qingping Yang, Chao Xin, Yu Yue, Lin Yan

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；SFT(abstract)；RLHF(abstract)

Optimizing Safe and Aligned Language Generation: A Multi-Objective GRPO Approach

Xuying Li, Zhuo Li, Yuji Kosuga, Victor Bian

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；preference optimization(abstract)

Empowering LLMs in Decision Games through Algorithmic Data Synthesis

Haolin Wang, Xueyan Li, Yazhe Niu, Shuai Hu, Hongsheng Li

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；post-training(abstract)

Comparative Analysis Based on DeepSeek, ChatGPT, and Google Gemini: Features, Techniques, Performance, Future Prospects

Anichur Rahman, Shahariar Hossain Mahir, Md Tanjum An Tashrif, Airin Afroj Aishi, Md Ahsan Karim, Dipanjali Kundu, Tanoy Debnath, Md. Abul Ala Moududi, MD. Zunead Abedin Eidmum

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Mask-DPO: Generalizable Fine-grained Factuality Alignment of LLMs

Yuzhe Gu, Wenwei Zhang, Chengqi Lyu, Dahua Lin, Kai Chen

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments Accepted by ICLR 2025. Code is available at https://github.com/open-compass/ANAH

CARMO: Dynamic Criteria Generation for Context-Aware Reward Modelling

Taneesh Gupta, Shivam Shandilya, Xuchao Zhang, Rahul Madhavan, Supriyo Ghosh, Chetan Bansal, Huaxiu Yao, Saravan Rajmohan

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；post-training(abstract)；RLHF(abstract)

Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown

Xingzhou Lou, Dong Yan, Wei Shen, Yuzi Yan, Jian Xie, Junge Zhang

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Optimizing Knowledge Integration in Retrieval-Augmented Generation with Self-Selection

Yan Weng, Fengbin Zhu, Tong Ye, Haoyan Liu, Fuli Feng, Tat-Seng Chua

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments 12 pages, 6 figures

Reveal the Mystery of DPO: The Connection between DPO and RL Algorithms

Xuerui Su, Yue Wang, Jinhua Zhu, Mingyang Yi, Feng Xu, Zhiming Ma, Yuting Liu

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；preference optimization(abstract)

Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis

Qining Zhang, Honghao Wei, Lei Ying

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Improving the Validity of Automatically Generated Feedback via Reinforcement Learning

Alexander Scarlatos, Digory Smith, Simon Woodhead, Andrew Lan

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments Best student paper award, Published in AIED 2024: The 25th International Conference on Artificial Intelligence in Education

Journal ref In International Conference on Artificial Intelligence in Education (pp. 280-294). Cham: Springer Nature Switzerland (2024)

UltraMedical: Building Specialized Generalists in Biomedicine

Kaiyan Zhang, Sihang Zeng, Ermo Hua, Ning Ding, Zhang-Ren Chen, Zhiyuan Ma, Haoxin Li, Ganqu Cui, Biqing Qi, Xuekai Zhu, Xingtai Lv, Hu Jinfang, Zhiyuan Liu, Bowen Zhou

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments Camera ready version for NeurIPS 2024 D&B Track

AlignCap: Aligning Speech Emotion Captioning to Human Preferences

Ziqi Liang, Haoxiang Shi, Hanhui Chen

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments Accepted to EMNLP2024 main conference

Generative Reward Models

Dakota Mahan, Duy Van Phung, Rafael Rafailov, Chase Blagden, Nathan Lile, Louis Castricato, Jan-Philipp Fränken, Chelsea Finn, Alon Albalak

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Minor DPO reject penalty to increase training robustness

Shiming Xie, Hong Chen, Fred Yu, Zeye Sun, Xiuyu Wu, Yingfan Hu

专题命中 后训练与偏好优化 ：LLM(abstract)；language model(abstract)；RLHF(abstract)；preference optimization(abstract)

Comments 8 pages, 19 figures

Critique-out-Loud Reward Models

Zachary Ankner, Mansheej Paul, Brandon Cui, Jonathan D. Chang, Prithviraj Ammanabrolu

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Robust Reinforcement Learning from Corrupted Human Feedback

Alexander Bukharin, Ilgee Hong, Haoming Jiang, Zichong Li, Qingru Zhang, Zixuan Zhang, Tuo Zhao

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；preference optimization(abstract)

Comments 22 pages, 7 figures

LLMs Beyond English: Scaling the Multilingual Capability of LLMs with Cross-Lingual Feedback

Wen Lai, Mohsen Mesgar, Alexander Fraser

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；instruction tuning(abstract)

Comments Accepted to Findings of ACL 2024. The code, datasets, and models are publicly available at https://github.com/boschresearch/ACL24-MLLM

Xwin-LM: Strong and Scalable Alignment Practice for LLMs

Bolin Ni, JingCheng Hu, Yixuan Wei, Houwen Peng, Zheng Zhang, Gaofeng Meng, Han Hu

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；SFT(abstract)；preference optimization(abstract)

Panacea: Pareto Alignment via Preference Adaptation for LLMs

Yifan Zhong, Chengdong Ma, Xiaoyuan Zhang, Ziran Yang, Haojun Chen, Qingfu Zhang, Siyuan Qi, Yaodong Yang

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Dense Reward for Free in Reinforcement Learning from Human Feedback

Alex J. Chan, Hao Sun, Samuel Holt, Mihaela van der Schaar

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

ICE-GRT: Instruction Context Enhancement by Generative Reinforcement based Transformers

Chen Zheng, Ke Sun, Da Tang, Yukun Ma, Yuyu Zhang, Chenguang Xi, Xun Zhou

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators

Andreas Liesenfeld, Alianda Lopez, Mark Dingemanse

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

AutoOR: Scalably Post-training LLMs to Autoformalize Operations Research Problems

AutoOR: 一种可扩展的后训练语言模型用于自动形式化运筹学问题

Sumeet Ramesh Motwani, Chuan Du, Aleksander Petrov, Christopher Davis, Philip Torr, Antonio Papania-Davis, Weishi Yan

机构 * The Moonshot Factory（谷歌登月工厂） ； University of Oxford（牛津大学） 纠错

专题命中 后训练与偏好优化 ：post-training(title,abstract)；分类 cs.AI、cs.LG

AI总结 AutoOR通过合成数据生成与强化学习管道，训练语言模型自动形式化运筹学问题，实现线性、混合整数和非线性优化问题的高效解决，取得六个经典运筹学基准测试的领先或竞争性结果。

SSPO: Structure-Aware Similarity-Weighted Preference Optimization for Neural Combinatorial Optimization

SSPO：面向神经组合优化的结构感知相似度加权偏好优化

Yuanyu Li, Jintao Xu, Zijiang Liu, Yongzhi Qi, Ningxuan Kang, Jianshen Zhang, Wei Qi, Chen Xie, Zuo-Jun Max Shen

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；分类 cs.AI、cs.LG

AI总结 该研究针对神经组合优化训练中的梯度信号极化与基线冗余问题，提出SSPO方法，通过结构感知相似度加权的留一法基线解决问题，其训练的EFL策略已在京东设施选址系统中实现实际应用。

Post-Training with Policy Gradients: Optimality and the Base Model Barrier

训练后使用策略梯度：最优性和基础模型障碍

Alireza Mousavi-Hosseini, Murat A. Erdogdu

专题命中 后训练与偏好优化 ：post-training(title,abstract)；分类 cs.AI、cs.LG

AI总结 本文研究了训练后使用策略梯度方法在序列预测中的最优性和基础模型限制，提出通过过程奖励模型克服支持范围障碍，并展示了自适应学习率的SGD和PG在统计学习和在线学习中的有效性。

Comments 39 pages, 2 figures. Published at the International Conference on Machine Learning (ICML) 2026

TELLER: Dual-Path Iterative Preference Optimization for Table Entity Linking

TELLER：用于表格实体链接的双路径迭代偏好优化

Yixin Peng, Kehao Li, Stefan Decker

机构 * RWTH Aachen（亚琛工业大学） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；分类 cs.CL、cs.LG

AI总结 该研究针对表格实体链接任务提出TELLER双路径迭代偏好优化方法，通过直接回答与推理两条路径分别优化，在TableInstruct和MammoTab V2数据集上提升了实体链接与推理生成的准确率。

Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL

掩码扩散语言模型是用于智能体强化学习的强大且可控的基于文本的世界模型

Darshan Deshpande

机构 * Patronus AI（守护神人工智能公司） 纠错

专题命中 后训练与偏好优化 ：language model(title,abstract)；分类 cs.AI、cs.LG

AI总结 研究强化学习中世界模型问题，对比自回归语言模型和掩码扩散语言模型，发现掩码扩散语言模型性能更优。引入GRPO训练框架，在三个OOD环境上零样本转移消融效果良好，还进行相关分析与评估，最后开源工作推动该领域研究。

Comments Dataset: https://huggingface.co/PatronusAI/world_model_corpus Training code: https://github.com/patronus-ai/mdlm_world_modeling

Never Too Late for Force: Accelerating VLA Post-Training with Reactive Force Injection

力，永不嫌迟：通过反应式力注入加速视觉-语言-动作模型的训练后优化

Yi Wang, Wendi Chen, Zimo Wen, Han Xue, Xueqi Li, Wenye Yu, Zhijie Chen, Hao Yang, Jun Lv, Chuan Wen, Cewu Lu

机构 * Shanghai Jiao Tong University（上海交通大学） ； Shanghai Innovation Institute（上海创新研究院） ； Southern University of Science and Technology（南方科技大学） ； Noematrix Ltd.（诺玛矩阵有限公司） 纠错

专题命中 后训练与偏好优化 ：post-training(title,abstract)；分类 cs.AI、cs.LG

AI总结 研究针对预训练VLA策略在接触状态下表现不佳的问题，提出LIFT框架，通过嫁接反应式动作专家、注入力及结合在线DAgger循环，提升其在富含接触操作任务中的性能，且证明相关组件对稳健操作很重要。

Comments Project page: https://lift-policy.github.io/