语言大模型 / LLM - arXivDaily 专题

2404.01054 2025-01-30 cs.CL cs.AI 88%

Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment

Yuu Jinnai, Tetsuro Morimura, Kaito Ariu, Kenshi Abe

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；RLHF(abstract)；preference optimization(abstract)

Comments NAACL 2025

2501.03486 2025-01-08 cs.LG cs.AI 88%

Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment

Prashant Trivedi, Souradip Chakraborty, Avinash Reddy, Vaneet Aggarwal, Amrit Singh Bedi, George K. Atia

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments 27 pages, Accepted in AAAI 2025

2409.06411 2024-12-02 cs.LG cs.CL 88%

Length Desensitization in Direct Preference Optimization

Wei Liu, Yang Bai, Chengcheng Han, Rongxiang Weng, Jun Xu, Xuezhi Cao, Jingang Wang, Xunliang Cai

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments 21 pages, 9 figures

2410.22304 2024-10-30 cs.CL cs.LG 88%

Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning

Yihe Deng, Paul Mineiro

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

Comments 5 pages, 4 figures, 1 table

2405.15230 2024-10-30 cs.AI cs.LG 88%

$i$REPO: $i$mplicit Reward Pairwise Difference based Empirical Preference Optimization

Long Tan Le, Han Shu, Tung-Anh Nguyen, Choong Seon Hong, Nguyen H. Tran

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

Comments Under Review

2410.20298 2024-10-29 cs.CL cs.AI 88%

Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data

Xinhong Xie, Tao Li, Quanyan Zhu

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

2410.20187 2024-10-29 cs.LG cs.AI stat.ML 88%

Uncertainty-Penalized Direct Preference Optimization

Sam Houliston, Alizée Pace, Alexander Immer, Gunnar Rätsch

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments Accepted at the NeurIPS 2024 FITML Workshop

2406.11715 2024-10-28 cs.LG cs.CL cs.SE 88%

Measuring memorization in RLHF for code completion

Aneesh Pappu, Billy Porter, Ilia Shumailov, Jamie Hayes

专题命中后训练与偏好优化：RLHF(title,abstract)；large language model(abstract)；language model(abstract)；preference optimization(abstract)

2409.03650 2024-10-04 cs.LG cs.CL 88%

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization

Yong Lin, Skyler Seto, Maartje ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, Tong Zhang

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract)；language model(abstract)；RLHF(abstract)

Comments 12 pages, 8 tables, 3 figures; Paper Accepted at EMNLP Findings 2024

2403.19159 2024-09-10 cs.CL cs.LG 88%

Disentangling Length from Quality in Direct Preference Optimization

Ryan Park, Rafael Rafailov, Stefano Ermon, Chelsea Finn

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

2407.12847 2024-07-19 cs.CL cs.AI cs.HC 88%

Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessments

Roland Daynauth, Jason Mars

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract)；large language model(abstract)；small language model(abstract)

2402.15018 2024-06-10 cs.CL cs.CY cs.LG 88%

Unintended Impacts of LLM Alignment on Global Representation

Michael J. Ryan, William Held, Diyi Yang

专题命中后训练与偏好优化：LLM(title)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments Accepted to ACL 2024 main conference

2402.06700 2024-06-07 cs.LG cs.AI 88%

Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement

Muning Wen, Junwei Liao, Cheng Deng, Jun Wang, Weinan Zhang, Ying Wen

专题命中后训练与偏好优化：language agent(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

2405.21040 2024-06-03 cs.CL cs.AI 88%

Direct Alignment of Language Models via Quality-Aware Self-Refinement

Runsheng Yu, Yong Wang, Xiaoqi Jiao, Youzhi Zhang, James T. Kwok

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract)；large language model(abstract)；RLHF(abstract)

2310.16836 2024-04-30 cs.CL cs.AI cs.AR cs.CV 88%

LLM-FP4: 4-Bit Floating-Point Quantized Transformers

Shih-yang Liu, Zechun Liu, Xijie Huang, Pingcheng Dong, Kwang-Ting Cheng

专题命中后训练与偏好优化：LLM(title,abstract)；large language model(abstract)；language model(abstract)；post-training(abstract)

Comments EMNLP 2023 Main Conference

Journal ref Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

2403.00199 2024-04-22 cs.CL cs.CY cs.LG 88%

Improving Socratic Question Generation using Data Augmentation and Preference Optimization

Nischal Ashok Kumar, Andrew Lan

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；prompting(abstract)

Comments Published at the 19th BEA Workshop co-located with NAACL-2024

2312.02554 2024-02-27 cs.LG cs.CL 88%

ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference

Tianchi Cai, Xierui Song, Jiyan Jiang, Fei Teng, Jinjie Gu, Guannan Zhang

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；RLHF(abstract)；preference optimization(abstract)

2311.12908 2023-11-23 cs.CV cs.AI cs.GR cs.LG 88%

Diffusion Model Alignment Using Direct Preference Optimization

Bram Wallace, Meihua Dang, Rafael Rafailov, Linqi Zhou, Aaron Lou, Senthil Purushwalkam, Stefano Ermon, Caiming Xiong, Shafiq Joty, Nikhil Naik

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

2310.12773 2023-10-20 cs.AI cs.LG 88%

Safe RLHF: Safe Reinforcement Learning from Human Feedback

Josef Dai, Xuehai Pan, Ruiyang Sun, Jiaming Ji, Xinbo Xu, Mickel Liu, Yizhou Wang, Yaodong Yang

专题命中后训练与偏好优化：RLHF(title,abstract)；LLM(abstract)；large language model(abstract)；language model(abstract)

2309.16240 2023-09-29 cs.LG cs.AI stat.ML 88%

Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints

Chaoqi Wang, Yibo Jiang, Chenghao Yang, Han Liu, Yuxin Chen

专题命中后训练与偏好优化：preference optimization(title,abstract)；large language model(abstract)；language model(abstract)；RLHF(abstract)

Comments Preprint

2308.08998 2023-08-22 cs.CL cs.LG 88%

Reinforced Self-Training (ReST) for Language Modeling

Caglar Gulcehre, Tom Le Paine, Srivatsan Srinivasan, Ksenia Konyushkova, Lotte Weerts, Abhishek Sharma, Aditya Siddhant, Alex Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey, Arnaud Doucet, Orhan Firat, Nando de Freitas

专题命中后训练与偏好优化：language model(title,abstract)；LLM(abstract)；large language model(abstract)；RLHF(abstract)

Comments 23 pages, 16 figures

2506.02018 2026-06-03 cs.CL 88%

Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data

增强释义类型生成：基于人工排序数据的DPO和RLHF评估影响

Christopher Lee Lübbers

机构 * University of Göttingen（哥廷根大学）

专题命中后训练与偏好优化：RLHF(title,title_cn)；language model(abstract)；preference optimization(abstract)；分类 cs.CL

AI总结本研究利用人工排序的释义类型数据集，结合直接偏好优化（DPO）使模型输出与人类判断对齐，将释义类型生成准确率提升3个百分点，人类偏好评分提升7个百分点，并创建了新的标注数据集以支持更严格的评估。

Comments 21 pages, 11 figures. Master's thesis, University of Goettingen, December 2024. Code: https://github.com/cluebbers/dpo-rlhf-paraphrase-types. Models: https://huggingface.co/collections/cluebbers/enhancing-paraphrase-type-generation-673ca8d75dfe2ce962a48ac0

2410.05451 2025-07-04 cs.CR cs.LG 88%

SecAlign: Defending Against Prompt Injection with Preference Optimization

Sizhe Chen, Arman Zharmagambetov, Saeed Mahloujifar, Kamalika Chaudhuri, David Wagner, Chuan Guo

机构 * UC Berkeley / Meta（伯克利大学/Meta）； Meta

专题命中后训练与偏好优化：preference optimization(title,abstract)；LLM(abstract,comments)；large language model(abstract)；language model(abstract)

Comments ACM CCS 2025. Key words: prompt injection defense, LLM security, LLM-integrated applications

2412.00967 2024-12-03 cs.AI 88%

Linear Probe Penalties Reduce LLM Sycophancy

Henry Papadatos, Rachel Freedman

专题命中后训练与偏好优化：LLM(title,abstract)；language model(abstract,comments)；large language model(abstract)；RLHF(abstract)

Comments 20 pages, 15 figures, NeurIPS 2024 Workshop Socially Responsible Language Modelling Research (SoLaR)

2304.05302 2023-10-10 cs.CL 88%

RRHF: Rank Responses to Align Language Models with Human Feedback without tears

Zheng Yuan, Hongyi Yuan, Chuanqi Tan, Wei Wang, Songfang Huang, Fei Huang

专题命中后训练与偏好优化：language model(title,abstract)；large language model(abstract)；SFT(abstract)；RLHF(abstract)

Comments ArXiv version For NeurIPS 2023 accepted paper: RRHF: Rank Responses to Align Language Models with Human Feedback

2608.08768 2026-08-11 cs.IR 新提交 88%

BOUND: Brief-Guided Corrective Preference Distillation at Search-Control Boundaries

BOUND：在搜索-控制边界处的摘要引导式校正偏好蒸馏

Qingying Niu, Ruiyang Ren, Wayne Xin Zhao, Yaliang Li

专题命中后训练与偏好优化：LLM(abstract,abstract_cn)；SFT(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结针对大语言模型深度搜索智能体的持续漂移问题，提出BOUND框架，通过摘要引导构建偏好对，用DPO蒸馏偏好，在7个基准上多数数据集和指标优于基线。

Comments 15 pages

2606.24937 2026-07-28 cs.AI cs.CL cs.IR cs.LG 版本更新 88%

The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

《智能体AI的搭车指南：从基础到系统》

Haggai Roitman

机构 * Haggai Roitman

专题命中后训练与偏好优化：LLM(abstract,abstract_cn)；SFT(abstract,abstract_cn)；RLHF(abstract,abstract_cn)；分类 cs.CL、cs.AI、cs.LG

AI总结本文为构建自主AI系统提供全面实践参考，覆盖从Transformer架构、训练优化到对齐、推理、检索增强生成、记忆系统、智能体设计模式及多智能体协调的完整技术栈。

Comments version 1.3

2607.20438 2026-07-24 cs.CL cs.AI cs.LG 新提交 88%

Preference Tuning as Spectral Update Reorganization

作为频谱更新重组的偏好调整

Peiyan Zhang, Haibo Jin, Liying Kang, Haohan Wang

机构 * University of Illinois at Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校）； Hong Kong Polytechnic University（香港理工大学）

专题命中后训练与偏好优化：RLHF(summary_cn,abstract)；post-training(abstract)；preference optimization(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结研究通过频谱结构研究RLHF及偏好优化，将偏好诱导更新转化为可干预对象，发现更新呈现头-尾组织，头部主导端点偏移，揭示了偏好训练后是结构化更新重组，以及对齐增益和覆盖损失与更新组织方式的关系。

2512.06343 2026-06-10 cs.LG cs.AI cs.CL 版本更新 88%

When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models

当距离干扰：BT损失中表示距离偏差对奖励模型的影响

Tong Xie, Andrew Bai, Yuanhao Ban, Yunqi Hong, Haoyu Li, Cho-Jui Hsieh

机构 * University of California, Berkeley（加州大学伯克利分校）

专题命中后训练与偏好优化：LLM(abstract,abstract_cn)；RLHF(abstract,abstract_cn)；large language model(abstract)；language model(abstract)

AI总结分析BT损失中表示距离导致的梯度偏差，提出NormBT自适应归一化方案，提升奖励模型在细粒度区分上的性能。

Comments ICML 2026

2601.14758 2026-05-29 cs.LG cs.AI cs.CL 88%

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models

从自回归到掩码扩散语言模型的后训练中的机制转变

Injin Kong, Hyoungjoon Lee, Yohan Jo

机构 * Graduate School of Data Science, Seoul National University（首尔国立大学数据科学研究生院）； Department of Biosystems & Biomaterials Science and Engineering, Seoul National University（首尔国立大学生物系统与生物材料科学与工程系）

专题命中后训练与偏好优化：post-training(title,abstract)；language model(title)；分类 cs.CL、cs.AI、cs.LG

AI总结通过比较电路分析，发现后训练得到的掩码扩散模型在结构上根据任务保留或重组自回归电路，在语义上从局部专业化转向分布式整合，表明扩散后训练是内部计算的深度重组。

1. 后训练与偏好优化 4516 篇

Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignment

Align-Pro: A Principled Approach to Prompt Optimization for LLM Alignment

Length Desensitization in Direct Preference Optimization

Flow-DPO: Improving LLM Mathematical Reasoning through Online Multi-Agent Learning

$i$REPO: $i$mplicit Reward Pairwise Difference based Empirical Preference Optimization

Learning from Response not Preference: A Stackelberg Approach for LLM Detoxification using Non-parallel Data

Uncertainty-Penalized Direct Preference Optimization

Measuring memorization in RLHF for code completion

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization

Disentangling Length from Quality in Direct Preference Optimization

Aligning Model Evaluations with Human Preferences: Mitigating Token Count Bias in Language Model Assessments

Unintended Impacts of LLM Alignment on Global Representation

Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement

Direct Alignment of Language Models via Quality-Aware Self-Refinement

LLM-FP4: 4-Bit Floating-Point Quantized Transformers

Improving Socratic Question Generation using Data Augmentation and Preference Optimization

ULMA: Unified Language Model Alignment with Human Demonstration and Point-wise Preference

Diffusion Model Alignment Using Direct Preference Optimization

Safe RLHF: Safe Reinforcement Learning from Human Feedback

Beyond Reverse KL: Generalizing Direct Preference Optimization with Diverse Divergence Constraints

Reinforced Self-Training (ReST) for Language Modeling

Enhancing Paraphrase Type Generation: The Impact of DPO and RLHF Evaluated with Human-Ranked Data

SecAlign: Defending Against Prompt Injection with Preference Optimization

Linear Probe Penalties Reduce LLM Sycophancy

RRHF: Rank Responses to Align Language Models with Human Feedback without tears

BOUND: Brief-Guided Corrective Preference Distillation at Search-Control Boundaries

The Hitchhiker's Guide to Agentic AI: From Foundations to Systems

Preference Tuning as Spectral Update Reorganization

When Distance Distracts: Representation Distance Bias in BT-Loss for Reward Models

Mechanism Shift During Post-training from Autoregressive to Masked Diffusion Language Models