语言大模型 / LLM

Why is Your Language Model a Poor Implicit Reward Model?

为什么你的语言模型是一个差的隐式奖励模型？

Noam Razin, Yong Lin, Jiarui Yao, Sanjeev Arora

机构 * Some Institute（某些研究所） 纠错

专题命中 后训练与偏好优化 ：language model(title,abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 研究发现隐式奖励模型（IM-RM）在泛化能力上劣于显式奖励模型（EX-RM），因其更依赖表面token级线索，而设计选择对模型泛化行为有显著影响。

Comments Accepted to ICLR 2026; Code available at https://github.com/princeton-pli/exrm-vs-imrm

The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes

LLM驱动的GUI代理的行为织体：人类价值观与交互结果

Simret Araya Gebreegziabher, Yukun Yang, Charles Chiang, Hojun Yoo, Chaoran Chen, Hyo Jin Do, Zahra Ashktorab, Werner Geyer, Diego Gómez-Zará, Toby Jia-Jun Li

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)

AI总结 本文研究了LLM驱动的GUI代理中人类价值观如何影响其行为和交互结果，提出了一种开源测试平台和实证分析方法。

Towards Native Intelligence: 6G-LLM Trained with Reinforcement Learning from NDT Feedback

迈向原生智能：通过NDT反馈强化学习训练的6G-LLM

Zhuoran Xiao, Tao Tao, Chenhui Ye, Yunbo Hu, Yijia Feng, Tianyu Jiao, Liyu Cai

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)

AI总结 本文提出通过NDT反馈强化学习训练6G-LLM，以提升网络调度精度和解决方案最优性。

Comments The paper has been accepted IEEE WCNC 2026

Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift

稳定偏好优化：一种针对灾难性偏好转移的双层方法

Chengtao Jian, Kai Yang, Tianhao Gao, Wuguang Ni, Keying Yang, Bowen Xiao, Jiajun Liu, Ye Ouyang

机构 * Tongji University（同济大学） ； AsiaInfo Technologies（亚洲信息科技） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 本文提出稳定偏好优化框架，通过约束偏好学习在安全对齐区域，解决灾难性偏好转移问题，提升偏好学习方法的稳定性和性能。

Contextual Biasing for LLM-Based ASR with Hotword Retrieval and Reinforcement Learning

基于上下文偏置的LLM语音识别中的热点词检索与强化学习

YuXiang Kong, JunFeng Hou, Jian Tang, Bingqing Zhu, Jicheng Zhang, Shaofei Xue

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)

AI总结 本文提出一种基于热点词检索与强化学习的LLM语音识别框架，通过增强数据增强和模糊匹配提高热点词识别准确性，从而降低关键词错误率并保持整体转录准确性。

Rethinking Direct Preference Optimization in Diffusion Models

重新思考扩散模型中的直接偏好优化

Junyong Kang, Seohyun Lim, Kyungjune Baek, Hyunjung Shim

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

AI总结 本文提出了一种改进扩散模型偏好优化的方法，通过稳定参考模型更新和时间步感知训练策略，提升模型在人类偏好评估中的性能。

Comments Accepted by SPIGM@NeurIPS 2025 and AAAI-26 (Oral)

Seedance 1.5 pro: A Native Audio-Visual Joint Generation Foundation Model

Seedance 1.5 pro: 一种原生音频视频联合生成基础模型

Team Seedance, Heyi Chen, Siyan Chen, Xin Chen, Yanfei Chen, Ying Chen, Zhuo Chen, Feng Cheng, Tianheng Cheng, Xinqi Cheng, Xuyan Chi, Jian Cong, Jing Cui, Qinpeng Cui, Qide Dong, Junliang Fan, Jing Fang, Zetao Fang, Chengjian Feng, Han Feng, Mingyuan Gao, Yu Gao, Dong Guo, Qiushan Guo, Boyang Hao, Qingkai Hao, Bibo He, Qian He, Tuyen Hoang, Ruoqing Hu, Xi Hu, Weilin Huang, Zhaoyang Huang, Zhongyi Huang, Donglei Ji, Siqi Jiang, Wei Jiang, Yunpu Jiang, Zhuo Jiang, Ashley Kim, Jianan Kong, Zhichao Lai, Shanshan Lao, Yichong Leng, Ai Li, Feiya Li, Gen Li, Huixia Li, JiaShi Li, Liang Li, Ming Li, Shanshan Li, Tao Li, Xian Li, Xiaojie Li, Xiaoyang Li, Xingxing Li, Yameng Li, Yifu Li, Yiying Li, Chao Liang, Han Liang, Jianzhong Liang, Ying Liang, Zhiqiang Liang, Wang Liao, Yalin Liao, Heng Lin, Kengyu Lin, Shanchuan Lin, Xi Lin, Zhijie Lin, Feng Ling, Fangfang Liu, Gaohong Liu, Jiawei Liu, Jie Liu, Jihao Liu, Shouda Liu, Shu Liu, Sichao Liu, Songwei Liu, Xin Liu, Xue Liu, Yibo Liu, Zikun Liu, Zuxi Liu, Junlin Lyu, Lecheng Lyu, Qian Lyu, Han Mu, Xiaonan Nie, Jingzhe Ning, Xitong Pan, Yanghua Peng, Lianke Qin, Xueqiong Qu, Yuxi Ren, Kai Shen, Guang Shi, Lei Shi, Yan Song, Yinglong Song, Fan Sun, Li Sun, Renfei Sun, Yan Sun, Zeyu Sun, Wenjing Tang, Yaxue Tang, Zirui Tao, Feng Wang, Furui Wang, Jinran Wang, Junkai Wang, Ke Wang, Kexin Wang, Qingyi Wang, Rui Wang, Sen Wang, Shuai Wang, Tingru Wang, Weichen Wang, Xin Wang, Yanhui Wang, Yue Wang, Yuping Wang, Yuxuan Wang, Ziyu Wang, Guoqiang Wei, Wanru Wei, Di Wu, Guohong Wu, Hanjie Wu, Jian Wu, Jie Wu, Ruolan Wu, Xinglong Wu, Yonghui Wu, Ruiqi Xia, Liang Xiang, Fei Xiao, XueFeng Xiao, Pan Xie, Shuangyi Xie, Shuang Xu, Jinlan Xue, Shen Yan, Bangbang Yang, Ceyuan Yang, Jiaqi Yang, Runkai Yang, Tao Yang, Yang Yang, Yihang Yang, ZhiXian Yang, Ziyan Yang, Songting Yao, Yifan Yao, Zilyu Ye, Bowen Yu, Jian Yu, Chujie Yuan, Linxiao Yuan, Sichun Zeng, Weihong Zeng, Xuejiao Zeng, Yan Zeng, Chuntao Zhang, Heng Zhang, Jingjie Zhang, Kuo Zhang, Liang Zhang, Liying Zhang, Manlin Zhang, Ting Zhang, Weida Zhang, Xiaohe Zhang, Xinyan Zhang, Yan Zhang, Yuan Zhang, Zixiang Zhang, Fengxuan Zhao, Huating Zhao, Yang Zhao, Hao Zheng, Jianbin Zheng, Xiaozheng Zheng, Yangyang Zheng, Yijie Zheng, Jiexin Zhou, Jiahui Zhu, Kuan Zhu, Shenhan Zhu, Wenjia Zhu, Benhui Zou, Feilong Zuo

专题命中 后训练与偏好优化 ：foundation model(title)；post-training(abstract)；SFT(abstract)；RLHF(abstract)

AI总结 Seedance 1.5 pro是一款专为原生音频视频联合生成设计的基础模型，通过双分支扩散变压器架构实现高质量生成，支持多语言唇形同步和动态摄像机控制，提升专业内容创作效率。

Comments Seedance 1.5 pro Technical Report

Preference Learning from Physics-Based Feedback: Tuning Language Models to Design BCC/B2 Superalloys

Satanu Ghosh, Collin Holgate, Neal R. Brodnik, Doug Downey, Samantha Daly, Tresa M. Pollock, Samuel Carton

机构 * Department of Computer Science, University of New Hampshire（新罕布什尔大学计算机科学系） ； Materials Department, University of California, Santa Barbara（加州大学圣芭芭拉分校材料系） ； Department of Mechanical Engineering, University of California, Santa Barbara（加州大学圣芭芭拉分校机械工程系） ； Allen Institute for Artificial Intelligence（人工智能研究院） 纠错

专题命中 后训练与偏好优化 ：language model(title,abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

Latent Principle Discovery for Language Model Self-Improvement

Keshav Ramji, Tahira Naseem, Ramón Fernandez Astudillo

机构 * IBM Research AI（IBM人工智能研究） 纠错

专题命中 后训练与偏好优化 ：language model(title,abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted at NeurIPS 2025

Teaching Pretrained Language Models to Think Deeper with Retrofitted Recurrence

Sean McLeish, Ang Li, John Kirchenbauer, Dayal Singh Kalra, Brian R. Bartoldson, Bhavya Kailkhura, Avi Schwarzschild, Jonas Geiping, Tom Goldstein, Micah Goldblum

机构 * University of Maryland（马里兰大学） ； New York University（纽约大学） ； Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室） ； University of North Carolina（北卡罗来纳大学） ； ELLIS Institute Tübingen, Max Planck Institute for Intelligent Systems, Tübingen AI Center（图宾根ELLIS研究所、马克斯·普朗克智能系统研究所、图宾根人工智能中心） ； Columbia University（哥伦比亚大学） 纠错

专题命中 后训练与偏好优化 ：language model(title,abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments code: https://github.com/mcleish7/retrofitting-recurrence, models: https://huggingface.co/collections/tomg-group-umd/retrofitting-recurrence

Better Estimation of the Kullback--Leibler Divergence Between Language Models

Afra Amini, Tim Vieira, Ryan Cotterell

机构 * ETH Zürich（苏黎世联邦理工学院） 纠错

专题命中 后训练与偏好优化 ：language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NeurIPS 2025

The Coverage Principle: How Pre-Training Enables Post-Training

Fan Chen, Audrey Huang, Noah Golowich, Sadhika Malladi, Adam Block, Jordan T. Ash, Akshay Krishnamurthy, Dylan J. Foster

专题命中 后训练与偏好优化 ：post-training(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models

Ziyi Wu, Anil Kag, Ivan Skorokhodov, Willi Menapace, Ashkan Mirzaei, Igor Gilitschenski, Sergey Tulyakov, Aliaksandr Siarohin

机构 * Snap Research ； University of Toronto（多伦多大学） ； Vector Institute（向量研究所） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；post-training(abstract)

Comments NeurIPS 2025 Spotlight. Project page: https://snap-research.github.io/DenseDPO/

Differential Information Distribution: A Bayesian Perspective on Direct Preference Optimization

Yunjae Won, Hyunji Lee, Hyeonbin Hwang, Minjoon Seo

机构 * KAIST AI（韩国科学技术院人工智能研究所） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint, under review. 39 pages, 12 figures. Updates from v1: Added new theoretical results on DPO training dynamics and policy exploration, included experiments with Qwen3-4B, and refined the discussion of log-margin dynamics

Visual Jigsaw Post-Training Improves MLLMs

Penghao Wu, Yushan Zhang, Haiwen Diao, Bo Li, Lewei Lu, Ziwei Liu

机构 * S-Lab, Nanyang Technological University（南洋理工大学S实验室） ； Linköping University（_linköping大学） ； SenseTime Research（商汤科技研究院） 纠错

专题命中 后训练与偏好优化 ：post-training(title,abstract)；large language model(abstract)；language model(abstract)

Captioning for Text-Video Retrieval via Dual-Group Direct Preference Optimization

Ji Soo Lee, Byungoh Ko, Jaewon Cho, Howoong Lee, Jaewoon Byun, Hyunwoo J. Kim

机构 * Korea University（韩国大学） ； Hanwha Vision（翰威英航） ； KAIST（韩国科学技术院） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Comments EMNLP 2025 Findings

Temporal Preference Optimization for Long-Form Video Understanding

Rui Li, Xiaohan Wang, Yuhui Zhang, Orr Zohar, Zeyu Wang, Serena Yeung-Levy

机构 * Stanford University（斯坦福大学） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

UAV Individual Identification via Distilled RF Fingerprints-Based LLM in ISAC Networks

Haolin Zheng, Ning Gao, Donghong Cai, Shi Jin, Michail Matthaiou

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)

VideoSAVi: Self-Aligned Video Language Models without Human Supervision

Yogesh Kulkarni, Pooyan Fazli

机构 * Arizona State University（亚利桑那州立大学） 纠错

专题命中 后训练与偏好优化 ：language model(title,abstract)；large language model(abstract)；preference optimization(abstract)

Comments COLM 2025

Sem-DPO: Mitigating Semantic Inconsistency in Preference Optimization for Prompt Engineering

Anas Mohamed, Azal Ahmad Khan, Xinran Wang, Ahmad Faraz Khan, Shuwen Ge, Saman Bahzad Khan, Ayaan Ahmad, Ali Anwar

机构 * University of Minnesota（明尼苏达大学） ； Virginia Tech（弗吉尼亚理工大学） ； Xi’an University of Technology（西安理工大学） ； Lahore University of Management Sciences（拉合尔管理科学大学） ； University of California, Santa Cruz（加州大学圣克鲁兹分校） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

DETONATE: A Benchmark for Text-to-Image Alignment and Kernelized Direct Preference Optimization

Renjith Prasad, Abhilekh Borah, Hasnat Md Abdullah, Chathurangi Shyalika, Gurpreet Singh, Ritvik Garimella, Rajarshi Roy, Harshul Surana, Nasrin Imanpour, Suranjana Trivedy, Amit Sheth, Amitava Das

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Comments 59 pages, 10 figures

Modality-Fair Preference Optimization for Trustworthy MLLM Alignment

Songtao Jiang, Yan Zhang, Ruizhe Chen, Tianxiang Hu, Yeying Jin, Qinglin He, Yang Feng, Jian Wu, Zuozhu Liu

机构 * Zhejiang University（浙江大学） ； ByteDance（字节跳动） ； National University of Singapore（新加坡国立大学） ； Angelalign Inc., China（中国Angelalign公司） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs

Xudong Li, Mengdan Zhang, Peixian Chen, Xiawu Zheng, Yan Zhang, Jingyuan Zheng, Yunhang Shen, Ke Li, Chaoyou Fu, Xing Sun, Rongrong Ji

机构 * Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学） ； Tencent Youtu Lab（腾讯优图实验室） ； Nanjing University（南京大学） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Reward Generalization in RLHF: A Topological Perspective

Tianyi Qiu, Fanzhi Zeng, Jiaming Ji, Dong Yan, Kaile Wang, Jiayi Zhou, Yang Han, Josef Dai, Xuehai Pan, Yaodong Yang

机构 * Center for AI Safety and Governance, Institute for AI, Peking University（人工智能安全与治理中心，人工智能研究院，北京大学） ； Tsinghua University（清华大学） ； Baichuan Inc（北川公司） 纠错

专题命中 后训练与偏好优化 ：RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 46 pages, ACL 2025 (Findings)

Fundamental Limits of Game-Theoretic LLM Alignment: Smith Consistency and Preference Matching

Zhekun Shi, Kaizhao Liu, Qi Long, Weijie J. Su, Jiancong Xiao

专题命中 后训练与偏好优化 ：LLM(title,abstract)；large language model(abstract)；language model(abstract)

Aligning Web Query Generation with Ranking Objectives via Direct Preference Optimization

João Coelho, Bruno Martins, João Magalhães, Chenyan Xiong

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Comments Accepted at SIGIR 2025

Inducing Robustness in a 2 Dimensional Direct Preference Optimization Paradigm

Sarvesh Shashidhar, Ritik, Nachiketa Patil, Suraj Racha, Ganesh Ramakrishnan

机构 * Centre for Machine Intelligence and Data Science, IIT Bombay（印度理工学院班加罗尔机器智能与数据科学中心） ； Koita Centre for Digital Health, IIT Bombay（印度理工学院班加罗尔数字健康中心） ； Department of Computer Science and Engineering, IIT Bombay（印度理工学院班加罗尔计算机科学与工程系） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title)；large language model(abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Updated abstract, algorithm and experimental results

Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimization

Noam Razin, Sadhika Malladi, Adithya Bhaskar, Danqi Chen, Sanjeev Arora, Boris Hanin

机构 * Princeton Language and Intelligence, Princeton University（普林斯顿语言与智能，普林斯顿大学） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to ICLR 2025; Code available at https://github.com/princeton-nlp/unintentional-unalignment

DSPO: Direct Semantic Preference Optimization for Real-World Image Super-Resolution

Miaomiao Cai, Simiao Li, Wei Li, Xudong Huang, Hanting Chen, Jie Hu, Yunhe Wang

机构 * University of Science and Technology of China（中国科学技术大学） ； Huawei Noah’s Ark Lab（华为诺亚实验室） 纠错

专题命中 后训练与偏好优化 ：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)

Auditing language models for hidden objectives

Samuel Marks, Johannes Treutlein, Trenton Bricken, Jack Lindsey, Jonathan Marcus, Siddharth Mishra-Sharma, Daniel Ziegler, Emmanuel Ameisen, Joshua Batson, Tim Belonax, Samuel R. Bowman, Shan Carter, Brian Chen, Hoagy Cunningham, Carson Denison, Florian Dietz, Satvik Golechha, Akbir Khan, Jan Kirchner, Jan Leike, Austin Meek, Kei Nishimura-Gasparian, Euan Ong, Christopher Olah, Adam Pearce, Fabien Roger, Jeanne Salle, Andy Shih, Meg Tong, Drake Thomas, Kelley Rivoire, Adam Jermyn, Monte MacDiarmid, Tom Henighan, Evan Hubinger

专题命中 后训练与偏好优化 ：language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG