大模型对齐与安全 - arXivDaily 专题

2601.16276 2026-01-26 cs.CL cs.AI cs.GT cs.LG cs.MA 67%

GameTalk: Training LLMs for Strategic Conversation

GameTalk: 训练 LLMs 进行战略性对话

Victor Conchello Vendrell, Max Ruiz Luyten, Mihaela van der Schaar

机构 * University of Cambridge（剑桥大学）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 GameTalk 通过多轮互动训练 LLMs 实现战略性决策，优于传统方法，尤其在奖励塑造下表现突出。

Comments 32 pages, 8 figures

2601.10128 2026-01-16 cs.CE cond-mat.mtrl-sci cs.SE 67%

A Generalizable Framework for Building Executable Domain-Specific LLMs under Data Scarcity: Demonstration on Semiconductor TCAD Simulation

在数据稀缺条件下构建可执行领域特定大语言模型的通用框架：以半导体TCAD仿真为例

Di Wang, Zhenhua Wu, Yu Liu, Kai Chang, Shaohua Wu

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

AI总结本文提出一种通用框架，在数据稀缺条件下构建可执行的领域特定大语言模型，通过生成合成数据和代码优化流程，实现领域知识灌输和脚本可执行性提升。

Comments Submitted to Nature Computational Science

2509.09681 2026-01-13 cs.IR cs.AI cs.CL cs.LG 67%

DB3 Team's Solution For Meta KDD Cup' 25

DB3团队的Meta KDD杯'25解决方案

Yikuan Xia, Jiazun Chen, Yirui Zhan, Suifeng Zhao, Weipeng Jiang, Chaorui Zhang, Wei Han, Bo Bai, Jun Gao

机构 * Key Laboratory of High Confidence Software Technologies, CS, Peking University, China（高可信软件技术重点实验室，中国科学院，北京大学）； Theory Lab, Central Research Institute, 2012 Labs, Huawei Technologies Co., Ltd, Shenzhen, China（理论实验室，中央研究院，2012实验室，华为技术有限公司，深圳）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 DB3团队通过多模态检索框架和拒绝训练方法，在KDD杯'25的CRAG-MM挑战中取得优异成绩，尤其在处理第一人称视角问题上表现突出。

2601.06543 2026-01-13 cs.CL cs.AI cs.LG 67%

SimLLM: Fine-Tuning Code LLMs for SimPy-Based Queueing System Simulation

SimLLM：用于基于SimPy的排队系统模拟的代码LLM微调

Jun-Qi Chen, Kun Zhang, Rui Zheng, Ying Zhong

机构 * Institute of Statistics and Big Data, Renmin University of China（中国人民大学统计与大数据研究院）； School of Information, Renmin University of China（中国人民大学信息学院）； School of Management and Economics, University of Electronic Science and Technology of China（电子科技大学管理学院）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 SimLLM通过微调开源LLM提升SimPy排队系统模拟代码生成能力，提供替代闭源模型的可行方案。

Comments 33 pages, 10 figures

2507.07723 2026-01-07 cs.AI cs.CL cs.LG 67%

Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift

稳定偏好优化：一种针对灾难性偏好转移的双层方法

Chengtao Jian, Kai Yang, Tianhao Gao, Wuguang Ni, Keying Yang, Bowen Xiao, Jiajun Liu, Ye Ouyang

机构 * Tongji University（同济大学）； AsiaInfo Technologies（亚洲信息科技）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本文提出稳定偏好优化框架，通过约束偏好学习在安全对齐区域，解决灾难性偏好转移问题，提升偏好学习方法的稳定性和性能。

2512.18014 2025-12-23 cs.CL cs.AI cs.LG 67%

ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India

ReGal：基于PPO的法律AI在印度判决预测和摘要中的初步探索

Shubham Kumar Nigam, Tanuj Tyagi, Siddharth Shukla, Aditya Kumar Guru, Balaramamahanthi Deepak Patnaik, Danush Khanna, Noel Shallum, Kripabandhu Ghosh, Arnab Bhattacharya

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 ReGal通过PPO结合多任务指令微调与强化学习，探索法律AI在印度判决预测和摘要中的应用，揭示了强化学习在法律文本中的挑战与潜力。

Comments Accepted in AILaw @ AAAI 2026 conference

2505.24850 2025-12-16 cs.LG cs.AI cs.CL 67%

Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning

利用负信号：从教师数据中进行强化蒸馏以提升大语言模型推理能力

Shuyao Xu, Cheng Peng, Jiangxuan Long, Weidi Xu, Wei Chu, Yuan Qi

机构 * National University of Singapore（国立新加坡大学）； INF AI

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本文提出通过强化蒸馏利用正负推理轨迹提升LLM推理性能，实验表明在少量数据下可达到与大量数据训练模型相当的性能。

Comments 22 pages, 10 figures. Code available at https://github.com/Tim-Siu/reinforcement-distillation

2511.03317 2025-12-03 cs.CV 67%

Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models

Diffusion-SDPO：用于扩散模型的受保护直接偏好优化

Minghao Fu, Guo-Hua Wang, Tianyu Cui, Qing-Guo Chen, Zhao Xu, Weihua Luo, Kaifu Zhang

机构 * School of Artificial Intelligence, Nanjing University（人工智能学院，南京大学）； National Key Laboratory for Novel Software Technology, Nanjing University（新型软件技术国家实验室，南京大学）； Alibaba International Digital Commerce Group（阿里巴巴国际数字商业集团）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

AI总结 Diffusion-SDPO通过保护性更新规则提升扩散模型对人类偏好的对齐效果。

Comments The code is publicly available at https://github.com/AIDC-AI/Diffusion-SDPO

2512.00042 2025-12-02 cs.CV cs.AI cs.CL cs.CY 67%

Closing the Gap: Data-Centric Fine-Tuning of Vision Language Models for the Standardized Exam Questions

弥合差距：面向标准化考试题目的视觉语言模型数据驱动微调

Egemen Sert, Şeyda Ertekin

机构 * organization= Department of Computer Engineering, Middle East Technical University (METU) , city= Ankara , country= Türkiye ； organization= METU-DTX Digital Transformation \& Innovation Centre, METU , city= Ankara , country= Türkiye

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI、cs.CY

AI总结本研究通过高质量数据和优化语法提升视觉语言模型在标准化考试题目的多模态推理性能，达到接近SOTA的水平。

2511.19523 2025-11-26 cs.CR 67%

EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answering

EAGER: 边缘对齐的LLM防御方法用于鲁棒、高效且准确的网络安全问答

Onat Gungor, Roshan Sood, Jiasheng Zhou, Tajana Rosing

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

AI总结 EAGER通过参数高效量化与领域偏好对齐，提升网络安全问答的鲁棒性、效率和准确性，降低对抗攻击成功率并优化响应延迟。

2511.12036 2025-11-18 cs.CE cond-mat.mtrl-sci cs.AI cs.CL cs.LG 67%

Preference Learning from Physics-Based Feedback: Tuning Language Models to Design BCC/B2 Superalloys

Satanu Ghosh, Collin Holgate, Neal R. Brodnik, Doug Downey, Samantha Daly, Tresa M. Pollock, Samuel Carton

机构 * Department of Computer Science, University of New Hampshire（新罕布什尔大学计算机科学系）； Materials Department, University of California, Santa Barbara（加州大学圣芭芭拉分校材料系）； Department of Mechanical Engineering, University of California, Santa Barbara（加州大学圣芭芭拉分校机械工程系）； Allen Institute for Artificial Intelligence（人工智能研究院）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

2410.21597 2025-11-14 cs.CL cs.AI cs.LG 67%

Reducing the Scope of Language Models

David Yunis, Siyu Huo, Chulaka Gunasekara, Danish Contractor

机构 * Toyota Technological Institute at Chicago（丰田技术研究所（芝加哥））； IBM（IBM公司）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Appears in AAAI 2026 in the Main Technical Track

2511.08394 2025-11-12 cs.CL cs.AI cs.HC cs.LG 67%

Interaction Dynamics as a Reward Signal for LLMs

Sian Gooding, Edward Grefenstette

机构 * Google DeepMind（谷歌DeepMind）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2510.25884 2025-10-31 cs.AI cs.CL cs.LG 67%

Approximating Human Preferences Using a Multi-Judge Learned System

Eitán Sprejer, Fernando Avalos, Augusto Bernardi, Jose Pedro Brito de Azevedo Faustino, Jacob Haimes, Narmeen Fatimah Oozeer

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2505.02820 2025-10-31 cs.AI cs.CL cs.LG 67%

AutoLibra: Agent Metric Induction from Open-Ended Human Feedback

Hao Zhu, Phil Cuvin, Xinkai Yu, Charlotte Ka Yee Yan, Jason Zhang, Diyi Yang

机构 * Stanford University（斯坦福大学）； University of Toronto（多伦多大学）； University of Pennsylvania（宾夕法尼亚大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments https://github.com/Open-Social-World/autolibra

2505.19601 2025-10-28 cs.LG cs.AI cs.CL 67%

Preference Optimization by Estimating the Ratio of the Data Distribution

Yeongmin Kim, Heesun Bae, Byeonghu Na, Il-Chul Moon

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NeurIPS 2025

2504.10637 2025-10-28 cs.CL cs.AI cs.LG 67%

Better Estimation of the Kullback--Leibler Divergence Between Language Models

Afra Amini, Tim Vieira, Ryan Cotterell

机构 * ETH Zürich（苏黎世联邦理工学院）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NeurIPS 2025

2505.11475 2025-10-27 cs.CL cs.AI cs.LG 67%

HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages

Zhilin Wang, Jiaqi Zeng, Olivier Delalleau, Hoo-Chang Shin, Felipe Soares, Alexander Bukharin, Ellie Evans, Yi Dong, Oleksii Kuchaiev

机构 * NVIDIA

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments NeurIPS 2025 Datasets and Benchmarks Track Camera Ready, 46 pages, 2 figures

2505.11080 2025-10-27 cs.CL cs.AI cs.LG 67%

BLEUBERI: BLEU is a surprisingly effective reward for instruction following

Yapei Chang, Yekyung Kim, Michael Krumdick, Amir Zadeh, Chuan Li, Chris Tanner, Mohit Iyyer

机构 * University of Maryland, College Park（马里兰大学学院公园分校）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments neurips cam-ready

2509.17999 2025-10-22 cs.CY cs.AI cs.HC cs.LG 67%

The Narcissus Hypothesis: Descending to the Rung of Illusion

Riccardo Cadei, Christian Internò

专题命中偏好对齐：alignment(abstract)；分类 cs.AI、cs.CY、cs.LG

Comments NeurIPS 2025 Workshop on Evaluating the Evolving LLM Lifecycle: Benchmarks, Emergent Abilities, and Scaling

2506.03517 2025-10-13 cs.CV 67%

DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models

Ziyi Wu, Anil Kag, Ivan Skorokhodov, Willi Menapace, Ashkan Mirzaei, Igor Gilitschenski, Sergey Tulyakov, Aliaksandr Siarohin

机构 * Snap Research ； University of Toronto（多伦多大学）； Vector Institute（向量研究所）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

Comments NeurIPS 2025 Spotlight. Project page: https://snap-research.github.io/DenseDPO/

2510.07557 2025-10-10 cs.LG cs.AI cs.CY cs.HC 67%

Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic

Abhay Bhandarkar, Gaurav Mishra, Khushi Juchani, Harsh Singhal

机构 * Ramaiah Institute of Technology（拉马亚院技术学院）； Ecofy Finance Private Limited（Ecofy金融私人有限公司）

专题命中偏好对齐：alignment(abstract)；分类 cs.AI、cs.CY、cs.LG

2506.08022 2025-10-10 cs.LG cs.AI cs.CL cs.CV 67%

Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining

Chenxi Liu, Tianyi Xiong, Yanshuo Chen, Ruibo Chen, Yihan Wu, Junfeng Guo, Tianyi Zhou, Heng Huang

机构 * University of Maryland, College Park（马里兰大学学院 park）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2503.03654 2025-10-09 cs.CL cs.AI cs.LG 67%

Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL

Jessica Hoffmann, Christiane Ahlheim, Zac Yu, Aria Walfrand, Jarvis Jin, Marie Tano, Ahmad Beirami, Erin van Liemt, Nithum Thain, Hakim Sidahmed, Lucas Dixon

机构 * Google DeepMind（谷歌DeepMind）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

2502.03323 2025-10-03 cs.CL cs.AI cs.LG 67%

Out-of-Distribution Detection using Synthetic Data Generation

Momin Abbas, Muneeza Azmat, Raya Horesh, Mikhail Yurochkin

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to COLM 2025. Camera-ready version

2509.26231 2025-10-01 cs.CV 67%

IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance

Jiayi Guo, Chuanhao Yan, Xingqian Xu, Yulin Wang, Kai Wang, Gao Huang, Humphrey Shi

机构 * Tsinghua University（清华大学）

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

Comments ICCV 2025

2509.25409 2025-10-01 cs.CL cs.AI cs.LG 67%

From Faithfulness to Correctness: Generative Reward Models that Think Critically

Qiyao Ma, Yunsheng Shi, Hongtao Tian, Chao Wang, Weiming Chang, Ting Yao

机构 * University of California, Davis（加州大学戴维斯分校）； WeChat, Tencent（微信、腾讯）； Tsinghua University（清华大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

2506.03006 2025-10-01 cs.SE 67%

A Preference-Driven Methodology for High-Quality Solidity Code Generation

Zhiyuan Peng, Xin Yin, Chenhao Ying, Chao Ni, Yuan Luo

专题命中偏好对齐：alignment(abstract)；DPO(abstract)

2505.13257 2025-09-30 cs.CL cs.AI cs.LG 67%

Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?

Zilu Tang, Afra Feyza Akyürek, Ekin Akyürek, Derry Wijaya

机构 * Boston University（波士顿大学）； MIT（麻省理工学院）； Monash University Indonesia（墨尔本大学印度尼西亚分校）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 9 pages, EMNLP PALS workshop 2025

2509.22633 2025-09-29 stat.ML cs.AI cs.CL cs.LG math.ST stat.TH 67%

Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback

Gen Li, Yuling Yan

机构 * Department of Statistics and Data Science, The Chinese University of Hong Kong（统计与数据科学系，香港中文大学）； Department of Statistics, University of Wisconsin-Madison（统计系，威斯康星大学麦迪逊分校）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL、cs.AI、cs.LG

1. 偏好对齐 3243 篇

GameTalk: Training LLMs for Strategic Conversation

A Generalizable Framework for Building Executable Domain-Specific LLMs under Data Scarcity: Demonstration on Semiconductor TCAD Simulation

DB3 Team's Solution For Meta KDD Cup' 25

SimLLM: Fine-Tuning Code LLMs for SimPy-Based Queueing System Simulation

Stable Preference Optimization: A Bilevel Approach to Catastrophic Preference Shift

ReGal: A First Look at PPO-based Legal AI for Judgment Prediction and Summarization in India

Harnessing Negative Signals: Reinforcement Distillation from Teacher Data for LLM Reasoning

Diffusion-SDPO: Safeguarded Direct Preference Optimization for Diffusion Models

Closing the Gap: Data-Centric Fine-Tuning of Vision Language Models for the Standardized Exam Questions

EAGER: Edge-Aligned LLM Defense for Robust, Efficient, and Accurate Cybersecurity Question Answering

Preference Learning from Physics-Based Feedback: Tuning Language Models to Design BCC/B2 Superalloys

Reducing the Scope of Language Models

Interaction Dynamics as a Reward Signal for LLMs

Approximating Human Preferences Using a Multi-Judge Learned System

AutoLibra: Agent Metric Induction from Open-Ended Human Feedback

Preference Optimization by Estimating the Ratio of the Data Distribution

Better Estimation of the Kullback--Leibler Divergence Between Language Models

HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages

BLEUBERI: BLEU is a surprisingly effective reward for instruction following

The Narcissus Hypothesis: Descending to the Rung of Illusion

DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models

Investigating Thematic Patterns and User Preferences in LLM Interactions using BERTopic

Modality-Balancing Preference Optimization of Large Multimodal Models by Adversarial Negative Mining

Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL

Out-of-Distribution Detection using Synthetic Data Generation

IMG: Calibrating Diffusion Models via Implicit Multimodal Guidance

From Faithfulness to Correctness: Generative Reward Models that Think Critically

A Preference-Driven Methodology for High-Quality Solidity Code Generation

Is Active Persona Inference Necessary for Aligning Small Models to Personal Preferences?

Towards Efficient Online Exploration for Reinforcement Learning with Human Feedback