语言大模型 / LLM

A Systematic Analysis of Base Model Choice for Reward Modeling

Kian Ahrabian, Pegah Jandaghi, Negar Mokhberian, Sai Praneeth Karimireddy, Jay Pujara

机构 * University of Southern California（南加州大学） ； Information Sciences Institute（信息科学研究所） 纠错

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；post-training(abstract)；RLHF(abstract)

Comments 19 pages, 13 figures, 5 tables

PLHF: Prompt Optimization with Few-Shot Human Feedback

Chun-Pai Yang, Kan Zheng, Shou-De Lin

机构 * ZRT Technology（ZRT技术） ； National Taiwan University（国立台湾大学） 纠错

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

OnRL-RAG: Real-Time Personalized Mental Health Dialogue System

Ahsan Bilal, Beiyu Lin

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments It needs more revisions. I am currently working on it with my co-author

Sample Efficient Preference Alignment in LLMs via Active Exploration

Viraj Mehta, Syrine Belakaria, Vikramjeet Das, Ojash Neopane, Yijia Dai, Ilija Bogunovic, Barbara Engelhardt, Stefano Ermon, Jeff Schneider, Willie Neiswanger

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；preference optimization(abstract)

Wasserstein Distances, Neuronal Entanglement, and Sparsity

Shashata Sawmya, Linghao Kong, Ilia Markov, Dan Alistarh, Nir Shavit

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；post-training(abstract)

Comments 10 pages, 9 figures

Pareto-Optimal Learning from Preferences with Hidden Context

Ryan Bahlous-Boldi, Li Ding, Lee Spector, Scott Niekum

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Align Anything: Training All-Modality Models to Follow Instructions with Language Feedback

Jiaming Ji, Jiayi Zhou, Hantao Lou, Boyuan Chen, Donghai Hong, Xuyao Wang, Wenqi Chen, Kaile Wang, Rui Pan, Jiahao Li, Mohan Wang, Josef Dai, Tianyi Qiu, Hua Xu, Dong Li, Weipeng Chen, Jun Song, Bo Zheng, Yaodong Yang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；post-training(abstract)；RLHF(abstract)

Imitate Before Detect: Aligning Machine Stylistic Preference for Machine-Revised Text Detection

Jiaqi Chen, Xiaoye Zhu, Tianyang Liu, Ying Chen, Xinhui Chen, Yiwen Yuan, Chak Tou Leong, Zuchao Li, Tang Long, Lei Zhang, Chenyu Yan, Guanghao Mei, Jie Zhang, Lefei Zhang

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments To appear at AAAI 2025. 14 pages, 6 figure

Enhancing LLMs for Physics Problem-Solving using Reinforcement Learning with Human-AI Feedback

Avinash Anand, Kritarth Prasad, Chhavi Kirtani, Ashwin R Nair, Mohit Gupta, Saloni Garg, Anurag Gautam, Snehal Buldeo, Rajiv Ratn Shah

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Word Alignment as Preference for Machine Translation

Qiyu Wu, Masaaki Nagata, Zhongtao Miao, Yoshimasa Tsuruoka

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments EMNLP 2024 Main

Enhancing Multiple Dimensions of Trustworthiness in LLMs via Sparse Activation Control

Yuxin Xiao, Chaoqun Wan, Yonggang Zhang, Wenxiao Wang, Binbin Lin, Xiaofei He, Xu Shen, Jieping Ye

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

sDPO: Don't Use Your Data All at Once

Dahyun Kim, Yungi Kim, Wonho Song, Hyeonwoo Kim, Yunsu Kim, Sanghoon Kim, Chanjun Park

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment

Yan Liu, Xiaoyuan Yi, Xiaokang Chen, Jing Yao, Jingwei Yi, Daoguang Zan, Zheng Liu, Xing Xie, Tsung-Yi Ho

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

OPTune: Efficient Online Preference Tuning

Lichang Chen, Jiuhai Chen, Chenxi Liu, John Kirchenbauer, Davit Soselia, Chen Zhu, Tom Goldstein, Tianyi Zhou, Heng Huang

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；preference optimization(abstract)

Comments 16 pages, 7 figures

Nash Learning from Human Feedback

Rémi Munos, Michal Valko, Daniele Calandriello, Mohammad Gheshlaghi Azar, Mark Rowland, Zhaohan Daniel Guo, Yunhao Tang, Matthieu Geist, Thomas Mesnard, Andrea Michi, Marco Selvi, Sertan Girgin, Nikola Momchev, Olivier Bachem, Daniel J. Mankowitz, Doina Precup, Bilal Piot

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Adaptive Preference Scaling for Reinforcement Learning with Human Feedback

Ilgee Hong, Zichong Li, Alexander Bukharin, Yixiao Li, Haoming Jiang, Tianbao Yang, Tuo Zhao

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；preference optimization(abstract)

Towards Analyzing and Understanding the Limitations of DPO: A Theoretical Perspective

Duanyu Feng, Bowen Qin, Chen Huang, Zheng Zhang, Wenqiang Lei

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；SFT(abstract)；preference optimization(abstract)

Comments Draft version

Using Human Feedback to Fine-tune Diffusion Models without Any Reward Model

Kai Yang, Jian Tao, Jiafei Lyu, Chunjiang Ge, Jiaxin Chen, Qimai Li, Weihan Shen, Xiaolong Zhu, Xiu Li

专题命中 后训练与偏好优化 ：large language model(abstract)；language model(abstract)；RLHF(abstract)；preference optimization(abstract)

Comments CVPR 2024 accepted; huggingface daily paper

HRLAIF: Improvements in Helpfulness and Harmlessness in Open-domain Reinforcement Learning From AI Feedback

Ang Li, Qiugen Xiao, Peng Cao, Jian Tang, Yi Yuan, Zijie Zhao, Xiaoyuan Chen, Liang Zhang, Xiangyang Li, Kaitong Yang, Weidong Guo, Yukang Gan, Xu Yu, Daniell Wang, Ying Shan

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments 18 pages, 7 figures

Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards

Wei Shen, Xiaoying Zhang, Yuanshun Yao, Rui Zheng, Hongyi Guo, Yang Liu

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Can LLMs Capture Human Preferences?

Ali Goli, Amandeep Singh

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberation

Ruixin Yang, Dheeraj Rajagopal, Shirley Anugrah Hayati, Bin Hu, Dongyeop Kang

专题命中 后训练与偏好优化 ：LLM(abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments Accepted at ICLR 2024 Workshop on Reliable and Responsible Foundation Models

Beyond Flat Policies: Hierarchical Post-Training for Embodied Agents in Robotic Manipulation

超越扁平策略：面向机器人操作具身智能体的分层后训练

He Kong, Zengjue Chen, Qi Wang, Qianli Xing, Runliang Niu, Peidong Liu, Jiawei Li, Shiqi Wang, Yi Chang

专题命中 后训练与偏好优化 ：post-training(title,abstract)；language model(abstract)

AI总结 针对现有VLA模型扁平策略难以处理长时程操作的问题，提出HiRoC分层后训练框架，通过解耦规划与执行并对齐分布，在机器人操作基准上取得优于强基线的性能。

Can Vision-Language Models Reason about AI Edits in Images?

视觉-语言模型能否对图像中的AI编辑进行推理？

Darsha Udayanga, Pin-Yu Chen, Payel Das, Qiang Ji

机构 * Rensselaer Polytechnic Institute（伦斯勒理工学院） ； IBM Research（IBM研究院） 纠错

专题命中 后训练与偏好优化 ：language model(title,abstract)

AI总结 本研究探究能否用强化学习而非显式推理监督训练视觉-语言模型推理AI图像编辑，提出基于GRPO的框架，引入eff-IoU指标，在多数据集上实现与SOTA相当的检测定位性能。

Temporal Concentration from Rollout Errors: Implicit Preference Optimization for Text-to-Video Diffusion

基于回退误差的时间集中：文本到视频扩散的隐式偏好优化

Henglin Liu, Fangyuan Kong, Jing Wang, Yizhou Lin, Nisha Huang, Chang Liu, Xintao Wang, Pengfei Wan, Kun Gai, Xiu Li

机构 * Tsinghua University（清华大学） ； Kuaishou Technology（快手科技） ； Sun Yat-sen University（中山大学） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；post-training(abstract)

AI总结 针对文本到视频扩散模型的时间稀疏伪影问题，本文提出集中式隐式偏好优化（cIPO）框架，通过回退误差推导隐式偏好信号，将优化集中于高误差片段，有效提升了视频的真实性与时间连贯性。

Comments project page: https://henglin-liu.github.io/cIPO_vis/

Post-training for Efficient Communication via Convention Formation

通过形成惯例实现高效通信的后训练

Yilun Hua, Evan Wang, Yoav Artzi

专题命中 后训练与偏好优化 ：post-training(title,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 本文通过后训练提升大语言模型在多轮交互中形成惯例的能力，设计了两项新基准测试，展示了模型在惯例形成任务上的显著提升。

Comments Accepted to COLM 2025

OracleAnalyser: Analysing Implicit Semantics of Oracle Bone Scripts through MLLMs with Post-training

OracleAnalyser：通过后训练的多模态大语言模型分析甲骨文的隐式语义

Zijia Song, Yelin Wang, Zhengyi Ma, Zitong Yu, Tianheng Wang, Jiahuan Zhang, Taorui Wang, Kaicheng Yu

机构 * National University of Defense Technology（国防科技大学） ； ShanghaiTech University（上海科技大学） ； Great Bay University（大湾区大学） ； Westlake University（西湖大学） 纠错

专题命中 后训练与偏好优化 ：post-training(title,abstract)；preference optimization(abstract)

AI总结 提出OracleAnalyser推理框架，通过多阶段后训练和稳定焦点偏好优化算法，在3B参数模型上超越更大规模模型，实现甲骨文分析。

ABACUS: Adapting Unified Foundation Model for Bridging Image Count Understanding and Generation

ABACUS: 自适应统一基础模型，桥接图像计数理解与生成

Anindya Mondal, Sauradip Nag, Anjan Dutta

机构 * University of Surrey（萨里大学） ； Simon Fraser University（西蒙菲莎大学） 纠错

专题命中 后训练与偏好优化 ：foundation model(title,abstract)；language model(abstract)

AI总结 提出ABACUS统一视觉语言模型，通过密度感知自适应缩放、边界感知计数策略和循环一致性GRPO，无需特定训练即可处理多种计数任务并生成计数忠实图像，在七个基准上达到最优。

Comments Under review, webpage: https://mondalanindya.github.io/ABACUS/

PhyGDPO: Physics-Aware Groupwise Direct Preference Optimization for Physically Consistent Text-to-Video Generation

PhyGDPO: 物理感知的分组直接偏好优化以实现物理一致的文本到视频生成

Yuanhao Cai, Kunpeng Li, Menglin Jia, Jialiang Wang, Junzhe Sun, Feng Liang, Weifeng Chen, Felix Juefei-Xu, Chu Wang, Ali Thabet, Xiaoliang Dai, Xuan Ju, Alan Yuille, Ji Hou

机构 * Meta Superintelligence Labs（Meta超智能实验室） ； Johns Hopkins University（约翰霍普金斯大学） ； Meta BizAI（Meta商业人工智能） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)

AI总结 提出PhyAugPipe构建大规模物理增强数据集PhyVidGen-135K，并设计PhyGDPO框架，利用分组Plackett-Luce模型和物理引导奖励机制，实现物理一致的文本到视频生成。

Comments ECCV 2026

PISCES: Annotation-free Text-to-Video Post-Training via Optimal Transport-Aligned Rewards

PISCES: 基于最优传输对齐奖励的无标注文本到视频后训练

Minh-Quan Le, Gaurav Mittal, Cheng Zhao, David Gu, Dimitris Samaras, Mei Chen

机构 * Microsoft（微软） ； Stony Brook University（石溪大学） 纠错

专题命中 后训练与偏好优化 ：post-training(title,abstract)；language model(abstract)

AI总结 提出PISCES算法，通过双重最优传输对齐奖励模块，无需人工标注即可提升文本生成视频的质量和语义对齐，在VBench上超越现有方法。

Comments Accepted to ICML 2026. Project page and code: https://roar-ai.github.io/pisces/