大模型对齐与安全 - arXivDaily 专题

2601.15395 2026-05-08 cs.CL cs.AI cs.HC 79%

Beyond Fixed Psychological Personas: State Beats Trait, but Language Models are State-Blind

超越固定心理人格：状态胜过特质，但语言模型是状态盲的

Tamunotonye Harry, Ivoline Ngong, Chima Nweke, Yuanyuan Feng, Joseph Near

机构 * University of Vermont（佛蒙特大学）； Independent Researcher（独立研究者）

专题命中偏好对齐：RLHF(abstract,abstract_cn)；alignment(abstract)；分类 cs.CL、cs.AI

AI总结本文通过Chameleon数据集发现语言模型对状态盲，而奖励模型对用户状态反应不一致，推动情感计算和个性化对话研究。

Comments Accepted to Findings of ACL 2026

2510.16340 2026-04-29 cs.CL cs.AI 79%

Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models

思考中的思考：评估后训练语言模型的推理能力

Pratham Singla, Shivank Garg, Ayush Singh, Ishan Garg, Ketan Suhaas Saichandran

机构 * Indian Institute of Technology Roorkee（印度理工学院罗奥克学院）； Boston University（波士顿大学）

专题命中偏好对齐：DPO(abstract,abstract_cn)；alignment(abstract)；分类 cs.CL、cs.AI

AI总结研究评估后训练语言模型的推理能力，通过三个核心能力评估其意识、泛化和推理与输出的对齐情况，发现强化学习模型在意识和泛化上优于SFT模型，但推理与输出对齐较差。

2504.01919 2026-04-29 cs.CL cs.AI 79%

Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation

弥合语言鸿沟：大型语言模型在机器翻译中的应用综述

Baban Gain, Dibyanayan Bandyopadhyay, Asif Ekbal, Trilok Nath Singh

机构 * Department of Computer Science and Engineering（计算机科学与工程系）； IIT Patna（印度理工学院帕纳布）

专题命中偏好对齐：RLHF(abstract,abstract_cn)；alignment(abstract)；分类 cs.CL、cs.AI

AI总结本文综述了大型语言模型在机器翻译中的应用，探讨了提示方法、参数高效微调、合成数据生成等技术，分析了低资源翻译中的挑战与对策，以及文档级和语篇级翻译方法的发展。

2604.23988 2026-04-28 cs.LG cs.AI 79%

Hindsight Preference Optimization for Financial Time Series Advisory

金融时间序列建议的 hindsight 偏好优化

Yanwei Cui, Guanghui Wang, Xing Zhang, Peiyang He, Ziyuan Li, Bing Zhu, Wei Qiu, Xusheng Wang, Zheng Yu, Anqi Xin

机构 * AWS Generative AI Innovation Center（AWS生成式AI创新中心）； HSBC Holdings Plc（汇丰控股有限公司）； HSBC Technology Center China（汇丰技术中心（中国））

专题命中偏好对齐：DPO(abstract,abstract_cn)；alignment(abstract)；分类 cs.AI、cs.LG

AI总结本文提出通过回顾性偏好优化方法，利用观测结果生成偏好对，提升语言模型在金融时间序列预测中的建议质量。

Comments Accepted at ICLR 2026 TSALM Workshop

2510.24235 2026-04-21 cs.LG cs.AI 79%

PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling

PaTaRM：通过偏好感知任务自适应奖励模型弥合成对和点状信号

Ai Jian, Jingqing Ruan, Xing Ma, Xiaoyun Zhang, Dailin Li, Weipeng Zhang, Ke Zeng, Xunliang Cai

机构 * Beijing University of Posts and Telecommunications（北京邮电大学）； Meituan（美团）； University of Chinese Academy of Sciences（中国科学院大学）

专题命中偏好对齐：RLHF(abstract,abstract_cn)；alignment(abstract)；分类 cs.AI、cs.LG

AI总结 PaTaRM通过偏好感知奖励机制实现成对和点状信号的弥合，无需显式评分标签，提升奖励模型的鲁棒性和任务适应性，实验显示在多个基准测试中表现优异。

Comments ACL Main

2604.07506 2026-04-21 cs.AI cs.CL 79%

ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework

ReflectRM: 通过统一判断框架内的自我反思提升生成奖励模型

Kai Qin, Liangxin Liu, Yu Liang, Longzheng Wang, Yan Wang, Yueyang Zhang, Long Xia, Zhiyuan Sun, Houde Liu, Daiting Shi

机构 * Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院）； Baidu Inc.（百度公司）

专题命中偏好对齐：RLHF(abstract,abstract_cn)；alignment(abstract)；分类 cs.CL、cs.AI

AI总结 ReflectRM通过自我反思评估分析质量，提升偏好建模，实验表明其在四个基准测试中平均准确率提升3.7%，并有效缓解位置偏差。

Comments Preprint

2508.15815 2026-04-21 cs.CL cs.AI cs.HC 79%

User-Assistant Bias in LLMs

大语言模型中的用户-助手偏见

Xu Pan, Jingxuan Fan, Zidi Xiong, Ely Hahami, Jorin Overwiening, Ziqian Xie

机构 * Harvard University（哈佛大学）； University of Texas Health Science Center at Houston（德克萨斯大学健康科学中心休斯顿分校）

专题命中偏好对齐：DPO(abstract,abstract_cn)；alignment(abstract)；分类 cs.CL、cs.AI

AI总结本文研究了大语言模型中因角色标签训练数据不一致导致的用户-助手偏见，通过UserAssist基准测试发现指令微调模型存在强用户偏见，而基础和推理模型接近中性，揭示了角色标签训练的潜在影响。

2604.14951 2026-04-17 cs.CV cs.AI cs.CL cs.MM 79%

RaTA-Tool: Retrieval-based Tool Selection with Multimodal Large Language Models

RaTA-Tool: 基于检索的多模态大语言模型工具选择

Gabriele Mattioli, Evelyn Turri, Sara Sarto, Lorenzo Baraldi, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

机构 * University of Modena and Reggio Emilia（摩德纳和雷吉奥艾米利亚大学）； University of Pisa（比萨大学）

专题命中偏好对齐：DPO(abstract,abstract_cn)；alignment(abstract)；分类 cs.CL、cs.AI

AI总结本文提出RaTA-Tool框架，利用多模态大语言模型将多模态查询转化为结构化任务描述，并通过语义丰富的工具描述检索合适工具，支持开放世界场景下的工具扩展。

Comments ICPR 2026

2510.26202 2026-04-14 cs.CL cs.AI 79%

What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

我的人类反馈中有什么？通过稀疏自编码器学习可解释的偏好数据描述

Rajiv Movva, Smitha Milli, Sewon Min, Emma Pierson

机构 * UC Berkeley（加州大学伯克利分校）； FAIR at Meta（Meta AI研究院）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI

AI总结本文提出WIMHF方法，通过稀疏自编码器解释人类反馈数据，揭示偏好数据中的人类可解释特征，发现不同数据集中的偏好差异及潜在不安全偏好，为数据筛选和个性化调整提供支持。

Comments ICLR 2026 (oral). v2 adds SAE ablations and robustness checks. Code: https://github.com/rmovva/wimhf; Demo: https://rajivmovva.com/demo-wimhf/

2506.17252 2026-03-10 cs.LG cs.AI 79%

Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization

适应性批级样本调度用于直接偏好优化

Zixuan Huang, Yikun Ban, Lean Fu, Xiaojie Li, Zhongxiang Dai, Jianxin Li, Deqing Wang

机构 * Beihang University（北京航空航天大学）； Bytedance China（字节跳动中国）； The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

AI总结本文提出SamS算法，通过动态调度训练样本提升DPO性能，无需修改核心算法，有效提高LLM对齐效果。

2512.06097 2025-12-09 cs.CL cs.AI 79%

Empathy by Design: Aligning Large Language Models for Healthcare Dialogue

设计中的共情：为医疗对话对齐大语言模型

Emre Umucu, Guillermina Solis, Leon Garza, Emilia Rivas, Beatrice Lee, Anantaa Kotal, Aritran Piplai

机构 * Department of Public Health Sciences, The University of Texas at El Paso, USA（公共卫生科学系，德克萨斯理工大学埃尔帕索分校）； Department of Nursing, The University of Texas at El Paso, USA（护理系，德克萨斯理工大学埃尔帕索分校）； Department of Rehabilitation Sciences, The University of Texas at El Paso, USA（康复科学系，德克萨斯理工大学埃尔帕索分校）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；trustworthy(abstract)；分类 cs.CL、cs.AI

AI总结本文提出基于DPO的对齐框架，通过优化医疗对话中事实准确性、语义连贯性和共情能力，提升AI助手在医疗场景中的可信度和人文关怀。

2509.21528 2025-09-29 cs.LG cs.AI 79%

Preemptive Detection and Steering of LLM Misalignment via Latent Reachability

Sathwik Karnik, Somil Bansal

机构 * Safe and Intelligent Autonomy Lab, Stanford University（斯坦福大学安全与智能自主实验室）

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.AI、cs.LG

2506.05748 2025-06-09 cs.LG cs.AI 79%

Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performance

Rudransh Agnihotri, Ananya Pandey

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；safety(abstract)；分类 cs.AI、cs.LG

2505.04388 2025-05-30 cs.CL cs.AI 79%

The Aloe Family Recipe for Open and Specialized Healthcare LLMs

Dario Garcia-Gasulla, Jordi Bayarri-Planas, Ashwin Kumar Gururajan, Enrique Lopez-Cuena, Adrian Tormos, Daniel Hinjos, Pablo Bernabeu-Perez, Anna Arias-Duart, Pablo Agustin Martin-Torres, Marta Gonzalez-Mallo, Sergio Alvarez-Napagao, Eduard Ayguadé-Parra, Ulises Cortés

机构 * Barcelona Supercomputing Center (BSC-CNS)（巴塞罗那超级计算中心（BSC-CNS））； Universitat Politècnica de Catalunya - Barcelona Tech (UPC)（加泰罗尼亚理工大学-巴塞罗那技术大学（UPC））

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；safety(abstract)；分类 cs.CL、cs.AI

Comments Follow-up work from arXiv:2405.01886

2503.00231 2025-03-04 cs.CL cs.AI 79%

Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking

Samar M. Magdy, Sang Yun Kwon, Fakhraddin Alwajih, Safaa Abdelfadil, Shady Shehata, Muhammad Abdul-Mageed

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Project GitHub page is accessible at: https://github.com/UBC-NLP/jawaher

2406.15599 2025-02-10 cs.LG cs.AI 79%

Pareto-Optimal Learning from Preferences with Hidden Context

Ryan Bahlous-Boldi, Li Ding, Lee Spector, Scott Niekum

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.AI、cs.LG

2409.19605 2025-02-04 cs.LG cs.CL 79%

The Crucial Role of Samplers in Online Direct Preference Optimization

Ruizhe Shi, Runlong Zhou, Simon S. Du

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments ICLR accepted

2410.08458 2025-02-03 cs.LG cs.CL 79%

Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both

Abhijnan Nath, Changsoo Jung, Ethan Seefried, Nikhil Krishnaswamy

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2402.01878 2025-01-28 cs.CL cs.LG 79%

LiPO: Listwise Preference Optimization through Learning-to-Rank

Tianqi Liu, Zhen Qin, Junru Wu, Jiaming Shen, Misha Khalman, Rishabh Joshi, Yao Zhao, Mohammad Saleh, Simon Baumgartner, Jialu Liu, Peter J. Liu, Xuanhui Wang

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments Accepted at NAACL 2025

2501.01544 2025-01-06 cs.LG cs.CL stat.ML 79%

Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information

Rasul Tutnov, Antoine Grosnit, Haitham Bou-Ammar

专题命中偏好对齐：alignment(abstract)；DPO(abstract)；safety(abstract)；分类 cs.CL、cs.LG

2409.06411 2024-12-02 cs.LG cs.CL 79%

Length Desensitization in Direct Preference Optimization

Wei Liu, Yang Bai, Chengcheng Han, Rongxiang Weng, Jun Xu, Xuezhi Cao, Jingang Wang, Xunliang Cai

专题命中偏好对齐：RLHF(abstract)；DPO(abstract)；harmlessness(abstract)；分类 cs.CL、cs.LG

Comments 21 pages, 9 figures

2409.03650 2024-10-04 cs.LG cs.CL 79%

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization

Yong Lin, Skyler Seto, Maartje ter Hoeve, Katherine Metcalf, Barry-John Theobald, Xuan Wang, Yizhe Zhang, Chen Huang, Tong Zhang

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments 12 pages, 8 tables, 3 figures; Paper Accepted at EMNLP Findings 2024

2409.17791 2024-09-27 cs.CL cs.AI 79%

Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness

Jian Li, Haojing Huang, Yujia Zhang, Pengfei Xu, Xi Chen, Rui Song, Lida Shi, Jingwen Wang, Hao Xu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

Comments Accepted at EMNLP 2024 Findings

2409.10571 2024-09-18 cs.LG cs.AI 79%

ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood

Ruoyu Wang, Jiachen Sun, Shaowei Hua, Quan Fang

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.AI、cs.LG

2403.19159 2024-09-10 cs.CL cs.LG 79%

Disentangling Length from Quality in Direct Preference Optimization

Ryan Park, Rafael Rafailov, Stefano Ermon, Chelsea Finn

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

2404.11999 2024-09-02 cs.CL cs.AI 79%

Token-level Direct Preference Optimization

Yongcheng Zeng, Guoqing Liu, Weiyu Ma, Ning Yang, Haifeng Zhang, Jun Wang

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2311.09641 2024-06-21 cs.AI cs.CL cs.CR cs.HC 79%

RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models

Jiongxiao Wang, Junlin Wu, Muhao Chen, Yevgeniy Vorobeychik, Chaowei Xiao

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI

2406.12182 2024-06-19 cs.CL cs.AI 79%

Aqulia-Med LLM: Pioneering Full-Process Open-Source Medical Language Models

Lulu Zhao, Weihao Zeng, Xiaofeng Shi, Hua Zhou, Donglin Hao, Yonghua Lin

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.AI

2406.10977 2024-06-18 cs.CL cs.AI 79%

Toward Optimal LLM Alignments Using Two-Player Games

Rui Zheng, Hongyi Guo, Zhihan Liu, Xiaoying Zhang, Yuanshun Yao, Xiaojun Xu, Zhaoran Wang, Zhiheng Xi, Tao Gui, Qi Zhang, Xuanjing Huang, Hang Li, Yang Liu

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.AI

Comments Our code is released at https://github.com/ruizheng20/gpo

2406.07657 2024-06-13 cs.LG cs.CL 79%

OPTune: Efficient Online Preference Tuning

Lichang Chen, Jiuhai Chen, Chenxi Liu, John Kirchenbauer, Davit Soselia, Chen Zhu, Tom Goldstein, Tianyi Zhou, Heng Huang

专题命中偏好对齐：alignment(abstract)；RLHF(abstract)；DPO(abstract)；分类 cs.CL、cs.LG

Comments 16 pages, 7 figures

1. 偏好对齐 3235 篇

Beyond Fixed Psychological Personas: State Beats Trait, but Language Models are State-Blind

Thinking About Thinking: Evaluating Reasoning in Post-Trained Language Models

Bridging the Linguistic Divide: A Survey on Leveraging Large Language Models for Machine Translation

Hindsight Preference Optimization for Financial Time Series Advisory

PaTaRM: Bridging Pairwise and Pointwise Signals via Preference-Aware Task-Adaptive Reward Modeling

ReflectRM: Boosting Generative Reward Models via Self-Reflection within a Unified Judgment Framework

User-Assistant Bias in LLMs

RaTA-Tool: Retrieval-based Tool Selection with Multimodal Large Language Models

What's In My Human Feedback? Learning Interpretable Descriptions of Preference Data

Adaptive Batch-Wise Sample Scheduling for Direct Preference Optimization

Empathy by Design: Aligning Large Language Models for Healthcare Dialogue

Preemptive Detection and Steering of LLM Misalignment via Latent Reachability

Efficient Online RFT with Plug-and-Play LLM Judges: Unlocking State-of-the-Art Performance

The Aloe Family Recipe for Open and Specialized Healthcare LLMs

Jawaher: A Multidialectal Dataset of Arabic Proverbs for LLM Benchmarking

Pareto-Optimal Learning from Preferences with Hidden Context

The Crucial Role of Samplers in Online Direct Preference Optimization

Simultaneous Reward Distillation and Preference Learning: Get You a Language Model Who Can Do Both

LiPO: Listwise Preference Optimization through Learning-to-Rank

Many of Your DPOs are Secretly One: Attempting Unification Through Mutual Information

Length Desensitization in Direct Preference Optimization

On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimization

Self-supervised Preference Optimization: Enhance Your Language Model with Preference Degree Awareness

ASFT: Aligned Supervised Fine-Tuning through Absolute Likelihood

Disentangling Length from Quality in Direct Preference Optimization

Token-level Direct Preference Optimization

RLHFPoison: Reward Poisoning Attack for Reinforcement Learning with Human Feedback in Large Language Models

Aqulia-Med LLM: Pioneering Full-Process Open-Source Medical Language Models

Toward Optimal LLM Alignments Using Two-Player Games

OPTune: Efficient Online Preference Tuning