语言大模型 / LLM - arXivDaily 专题

2504.03622 2026-02-04 cs.CL cs.AI cs.LG 90%

Align to Structure: Aligning Large Language Models with Structural Information

对齐结构：将大型语言模型与结构信息对齐

Zae Myung Kim, Anand Ramachandran, Farideh Tavazoee, Joo-Kyung Kim, Oleg Rokhlenko, Dongyeop Kang

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结通过整合语言学话语框架到强化学习中，结构对齐方法提升LLMs在长文本生成和摘要任务中的连贯性和结构组织能力。

Comments Accepted to AAAI 2026 AIA

2512.13699 2025-12-17 cs.CY 90%

Us-vs-Them bias in Large Language Models

大语言模型中的‘我们 vs 他们’偏见

Tabia Tanzin Prama, Julia Witte Zimmerman, Christopher M. Danforth, Peter Sheridan Dodds

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；preference optimization(abstract)

AI总结本研究发现大语言模型在不同人设下表现出‘我们 vs 他们’偏见，通过微调和DPO方法可有效缓解这种偏见。

2511.12596 2025-11-18 cs.CL cs.AI cs.LG 90%

Group-Aware Reinforcement Learning for Output Diversity in Large Language Models

Oron Anschel, Alon Shoshan, Adam Botach, Shunit Haviv Hakimi, Asaf Gendler, Emanuel Ben Baruch, Nadav Bhonker, Igor Kviatkovsky, Manoj Aggarwal, Gerard Medioni

机构 * Amazon（亚马逊）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments EMNLP Main 2025

2510.21090 2025-10-27 cs.CL cs.AI cs.LG 90%

Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only

Qingru Zhang, Liang Qiu, Ilgee Hong, Zhenghao Xu, Tianyi Liu, Shiyang Li, Rongzhi Zhang, Zheng Li, Lihong Li, Bing Yin, Chao Zhang, Jianshu Chen, Haoming Jiang, Tuo Zhao

机构 * Georgia Institute of Technology（佐治亚理工学院）； Amazon（亚马逊）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；SFT(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by COLM 2025

2410.23022 2025-10-27 cs.LG cs.AI cs.CL cs.RO 90%

Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback

Qinqing Zheng, Mikael Henaff, Amy Zhang, Aditya Grover, Brandon Amos

机构 * \dagger（机构1）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments RLC 2025

2506.18369 2025-10-13 cs.CV 90%

RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models

Yeongtak Oh, Dohyun Chung, Juhyeon Shin, Sangha Park, Johan Barthelemy, Jisoo Mok, Sungroh Yoon

专题命中后训练与偏好优化：language model(title,abstract)；post-training(title,abstract)；large language model(abstract)；SFT(abstract)

Comments Accepted to NeurIPS 2025

2507.01915 2025-07-03 cs.CL cs.AI cs.LG 90%

Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models

Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, Qing He

机构 * Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (CAS)（智能信息处理重点实验室，计算技术研究所，中国科学院）； State Key Lab of AI Safety（人工智能安全国家重点实验室）； University of Chinese Academy of Sciences, CAS（中国科学院大学）； Zhejiang University（浙江大学）

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 19 pages, 3 figures. Accepted by ACL 2025 (main)

2407.07880 2025-04-21 cs.LG cs.AI cs.CL 90%

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

Junkang Wu, Yuexiang Xie, Zhengyi Yang, Jiancan Wu, Jiawei Chen, Jinyang Gao, Bolin Ding, Xiang Wang, Xiangnan He

专题命中后训练与偏好优化：language model(title,abstract)；preference optimization(title,abstract)；large language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.10596 2025-03-04 cs.CL cs.AI cs.LG 90%

Post-training an LLM for RAG? Train on Self-Generated Demonstrations

Matthew Finlayson, Ilia Kulikov, Daniel M. Bikel, Barlas Oguz, Xilun Chen, Aasish Pappu

专题命中后训练与偏好优化：LLM(title,abstract)；post-training(title)；large language model(abstract)；language model(abstract)

2502.04354 2025-02-10 cs.CL cs.AI cs.LG 90%

Reviving The Classics: Active Reward Modeling in Large Language Model Alignment

Yunyi Shen, Hao Sun, Jean-François Ton

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2412.18171 2024-12-30 cs.CR 90%

Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models

Xiaomeng Hu, Pin-Yu Chen, Tsung-Yi Ho

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；RLHF(abstract)

Comments Accepted by AAAI 2025. Project page: https://huggingface.co/spaces/TrustSafeAI/Token-Highlighter

2412.16451 2024-12-24 cs.LG cs.AI cs.CL 90%

Correcting Large Language Model Behavior via Influence Function

Han Zhang, Zhuo Zhang, Yi Zhang, Yuanzhao Zhai, Hanyang Peng, Yu Lei, Yue Yu, Hui Wang, Bin Liang, Lin Gui, Ruifeng Xu

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；post-training(abstract)；分类 cs.CL、cs.AI、cs.LG

2411.14797 2024-11-25 cs.LG cs.AI cs.CL cs.CV 90%

Continual SFT Matches Multimodal RLHF with Negative Supervision

Ke Zhu, Yu Wang, Yanpeng Sun, Qiang Chen, Jiangjiang Liu, Gang Zhang, Jingdong Wang

专题命中后训练与偏好优化：SFT(title,abstract)；RLHF(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2405.19544 2024-11-25 cs.AI cs.CL cs.LG math.OC stat.ML 90%

One-Shot Safety Alignment for Large Language Models via Optimal Dualization

Xinmeng Huang, Shuo Li, Edgar Dobriban, Osbert Bastani, Hamed Hassani, Dongsheng Ding

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 32 pages, 6 figures, 8 tables

2305.18290 2024-07-31 cs.LG cs.AI cs.CL 90%

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, Chelsea Finn

专题命中后训练与偏好优化：language model(title,abstract)；preference optimization(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2406.02756 2024-06-06 cs.CL cs.AI cs.LG 90%

Aligning Large Language Models via Fine-grained Supervision

Dehong Xu, Liang Qiu, Minseok Kim, Faisal Ladhak, Jaeyoung Do

专题命中后训练与偏好优化：language model(title,abstract)；large language model(title)；LLM(abstract)；RLHF(abstract)

2405.21046 2024-06-03 cs.LG cs.AI cs.CL stat.ML 90%

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

Tengyang Xie, Dylan J. Foster, Akshay Krishnamurthy, Corby Rosset, Ahmed Awadallah, Alexander Rakhlin

专题命中后训练与偏好优化：RLHF(title,abstract)；preference optimization(title,abstract)；language model(abstract)；分类 cs.CL、cs.AI、cs.LG

2402.10958 2024-05-29 cs.CL cs.AI cs.LG 90%

Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts

Yueqin Yin, Zhendong Wang, Yi Gu, Hai Huang, Weizhu Chen, Mingyuan Zhou

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(title)；large language model(abstract)；language model(abstract)

2404.09248 2024-04-16 cs.LG cs.AI cs.CL 90%

Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts

Jing-Cheng Pang, Si-Hang Yang, Kaiyuan Li, Jiaji Zhang, Xiong-Hui Chen, Nan Tang, Yang Yu

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

2403.09798 2024-03-18 cs.CY 90%

Comparing Rationality Between Large Language Models and Humans: Insights and Open Questions

Dana Alsagheer, Rabimba Karanjai, Nour Diallo, Weidong Shi, Yang Lu, Suha Beydoun, Qiaoning Zhang

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；RLHF(abstract)

2402.19464 2024-03-01 cs.LG cs.AI cs.CL 90%

Curiosity-driven Red-teaming for Large Language Models

Zhang-Wei Hong, Idan Shenfeld, Tsun-Hsuan Wang, Yung-Sung Chuang, Aldo Pareja, James Glass, Akash Srivastava, Pulkit Agrawal

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；LLM(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Published at ICLR 2024

2402.07282 2024-02-14 cs.CL cs.AI cs.LG 90%

How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?

Ryan Liu, Theodore R. Sumers, Ishita Dasgupta, Thomas L. Griffiths

专题命中后训练与偏好优化：large language model(title,abstract)；language model(title,abstract)；prompting(abstract)；分类 cs.CL、cs.AI、cs.LG

2608.16553 2026-08-18 cs.CL 新提交 90%

STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment

STAGE：面向多偏好大语言模型对齐的可控目标准入

Yongqi Tong, Zhenyu Zhang, Ruirui Wang, Kewei Fu, Shaoqing Lin, Sijie Dong, Jiang-Ming Yang, Xin Zhang, Jianshe Li

机构 * Ant International（蚂蚁国际）

专题命中后训练与偏好优化：LLM(title,summary_cn)；RLHF(summary_cn,abstract)；分类 cs.CL

AI总结该研究针对多偏好LLM对齐的目标准入时机问题，提出稳定性引导的STAGE控制器，通过活动集扩展与探测排序等方法，在多偏好对齐任务中取得优于基线的自动评估结果，为RLHF提供新控制变量。

2606.06586 2026-08-10 cs.CL 版本更新 90%

PolyFact: Comparing Consistency-Driven Post-training Methods for Cross-Lingual Factual Recall

通过一致性驱动的强化学习改进跨语言事实回忆

Jonathan von Rad, Louis Arts, George Burgess, Eleftheria Kolokytha, Harry O'Donnell, Ektor Oikonomidis Doumpas, Eduardo Sanchez, Yao Lu, Pontus Stenetorp

机构 * University College London（伦敦大学学院）； Centre for Artificial Intelligence（人工智能中心）

专题命中后训练与偏好优化：post-training(title,abstract)；SFT(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结提出PolyFact数据集，利用GRPO强化学习方法提升大语言模型的跨语言事实回忆一致性，优于监督微调，并揭示其通过减少语言专用表示实现跨语言共享的机制。

Comments Under Review at EACL 2026

2608.06246 2026-08-07 cs.LG 新提交 90%

A Six-Dimensional Taxonomy of Post-Training Adaptation Techniques with Applications in AI Governance

用于人工智能治理的后训练适应技术的六维分类法

Fardin Afdideh, Fernando Seoane, Farhad Abtahi

专题命中后训练与偏好优化：post-training(title,abstract)；large language model(abstract)；language model(abstract)；foundation model(abstract)

AI总结本综述构建了后训练适应技术的六维分类法，梳理技术间关系，为AI治理提供术语支持，并指出该领域的开放挑战。

2608.05886 2026-08-07 cs.SE cs.AI 新提交 90%

CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents

CodeGrep：用于LLM编码智能体的基于强化学习训练的检索智能体

Wuya Chen, Yihao yang, Yang Cao, Yue Lin

专题命中后训练与偏好优化：LLM(title,title_cn)；分类 cs.AI

AI总结该研究针对LLM编码智能体查找文件效率低的问题，提出基于GRPO训练的14B检索智能体CodeGrep，在SWE-Bench Verified上保持解决率的同时，减少了交互轮数和token使用量。

2607.23802 2026-08-03 cs.AI 版本更新 90%

From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

从可验证奖励强化学习到自验证奖励强化学习：任务转换为开放式语言模型自我改进带来自验证奖励

Qinsi Wang, Jing Shi, Huazheng Wang, Kun Wan, Yiran Wu, Bo Liu, Qingyun Wu, Hai Helen Li, Yiran Chen, Handong Zhao, Wentian Zhao

机构 * Duke University（杜克大学）； Adobe Inc.（奥多比公司）； Oregon State University（俄勒冈州立大学）； Pennsylvania State University（宾夕法尼亚州立大学）； National University of Singapore（新加坡国立大学）； Amazon（亚马逊）

专题命中后训练与偏好优化：LLM(title,summary_cn)；large language model(abstract)；language model(abstract)；分类 cs.AI

AI总结研究针对开放式LLM自我改进中奖励验证问题，提出基于任务转换的RLSVR方法，通过SpyRL实例化，在文本摘要等任务实验中，该方法在不可验证任务上优于现有方法，在可验证推理任务上有提升，扩展了基于RLVR的自我改进。

Comments COLM 2026

2607.16195 2026-07-23 cs.AI 版本更新 90%

Rater State Bias in RLHF Preference Data: An Audit Framework

强化学习从人类反馈（RLHF）中的评分者状态偏差：一个审计框架

Elena Kopteva, Vitaliy Hlynianyi-Zhuk

机构 * Oles Honchar Dnipro National University（第聂伯罗国立奥列斯·冈察尔大学）； Kyiv Institute of Modern Psychology and Psychotherapy（基辅现代心理与心理治疗研究所）

专题命中后训练与偏好优化：RLHF(title,title_cn)；分类 cs.AI

AI总结研究RLHF中评分者状态偏差这一结构化混淆，提出评分者状态变化是偏差来源，开发审计框架，定义相关概念，得出可证伪预测和效应大小阈值，提出审计协议和试点研究计划以分离偏差来源。

Comments v2: added sycophancy related work; Section 4 positioned against concurrent Bradley-Terry amplification results (Shapira et al. 2026); minor revisions

2607.18258 2026-07-22 cs.AI 新提交 90%

S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF

S2T-RLHF：基于稳定偏好的强化学习从人类反馈中的分层信用分配

Wei Chen, Guanghui Zhu, Yafei Li, Limin Wang, Yihua Huang

机构 * State Key Laboratory for Novel Software Technology, Nanjing University（南京大学计算机软件新技术国家重点实验室）； School of Computer Science and Artificial Intelligence, Zhengzhou University（郑州大学计算机科学与人工智能学院）

专题命中后训练与偏好优化：RLHF(title,title_cn)；分类 cs.AI

AI总结研究基于偏好奖励模型的RLHF训练不稳定问题，提出分层信用分配的粒度感知原则，引入S2T-RLHF框架，先在句子间分配奖励，再在句内细化，无需重新训练模型，实验证明该方法提高了训练稳定性和鲁棒性。

2603.17673 2026-07-07 cs.CR cs.AI 版本更新 90%

Towards Reliable Local Security Agents: Verifiable Post-Training for Linux Privilege Escalation

在Linux提权中使用可验证奖励进行训练后的本地LLM代理

Philipp Normann, Andreas Happe, Jürgen Cito, Daniel Arp

专题命中后训练与偏好优化：LLM(title_cn,abstract)；post-training(title,abstract)；language model(abstract)；分类 cs.AI

AI总结本文提出一种两阶段训练流程，通过监督微调和强化学习提升Linux提权任务的性能，实现高成功率和低推理成本。

1. 后训练与偏好优化 4507 篇

Align to Structure: Aligning Large Language Models with Structural Information

Us-vs-Them bias in Large Language Models

Group-Aware Reinforcement Learning for Output Diversity in Large Language Models

Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only

Online Intrinsic Rewards for Decision Making Agents from Large Language Model Feedback

RePIC: Reinforced Post-Training for Personalizing Multi-Modal Language Models

Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models

Towards Robust Alignment of Language Models: Distributionally Robustifying Direct Preference Optimization

Post-training an LLM for RAG? Train on Self-Generated Demonstrations

Reviving The Classics: Active Reward Modeling in Large Language Model Alignment

Token Highlighter: Inspecting and Mitigating Jailbreak Prompts for Large Language Models

Correcting Large Language Model Behavior via Influence Function

Continual SFT Matches Multimodal RLHF with Negative Supervision

One-Shot Safety Alignment for Large Language Models via Optimal Dualization

Direct Preference Optimization: Your Language Model is Secretly a Reward Model

Aligning Large Language Models via Fine-grained Supervision

Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHF

Relative Preference Optimization: Enhancing LLM Alignment through Contrasting Responses across Identical and Diverse Prompts

Knowledgeable Agents by Offline Reinforcement Learning from Large Language Model Rollouts

Comparing Rationality Between Large Language Models and Humans: Insights and Open Questions

Curiosity-driven Red-teaming for Large Language Models

How do Large Language Models Navigate Conflicts between Honesty and Helpfulness?

STAGE: Controlled Objective Admission for Multi-Preference LLM Alignment

PolyFact: Comparing Consistency-Driven Post-training Methods for Cross-Lingual Factual Recall

A Six-Dimensional Taxonomy of Post-Training Adaptation Techniques with Applications in AI Governance

CodeGrep: An RL-Trained Retrieval Agent for LLM Coding Agents

From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement

Rater State Bias in RLHF Preference Data: An Audit Framework

S2T-RLHF: Hierarchical Credit Assignment for Stable Preference-Based RLHF

Towards Reliable Local Security Agents: Verifiable Post-Training for Linux Privilege Escalation