大模型对齐与安全 - arXivDaily 专题

2510.19296 2025-12-09 cs.LG cs.AR cs.PL 57%

QiMeng-SALV: Signal-Aware Learning for Verilog Code Generation

QiMeng-SALV：面向Verilog代码生成的信号感知学习

Yang Zhang, Rui Zhang, Jiaming Guo, Lei Huang, Di Huang, Yunpu Zhao, Shuyao Cheng, Pengwei Jin, Chongxiao Li, Zidong Du, Xing Hu, Qi Guo, Yunji Chen

机构 * State Key Lab of Processors, Institute of Computing Technology, CAS（处理器国家重点实验室，计算技术研究所，中国科学院）； University of Chinese Academy of Sciences（中国科学院大学）； University of Science and Technology of China（中国科学技术大学）

专题命中偏好对齐：DPO(abstract)；分类 cs.LG

AI总结 QiMeng-SALV通过信号感知学习提升Verilog代码生成的准确性与性能，采用信号级优化解决功能奖励不足问题。

Comments Accepted to NeurIPS 2025

2412.08519 2025-12-09 cs.CL 57%

Bridging Relevance and Reasoning: Rationale Distillation in Retrieval-Augmented Generation

弥合相关性与推理：检索增强生成中的推理蒸馏

Pengyue Jia, Derong Xu, Xiaopeng Li, Zhaocheng Du, Xiangyang Li, Yichao Wang, Yuhao Wang, Qidong Liu, Maolin Wang, Huifeng Guo, Ruiming Tang, Xiangyu Zhao

机构 * City University of Hong Kong（香港城市大学）； University of Science and Technology of China（中国科学技术大学）； Huawei Noah’s Ark Lab（华为诺亚实验室）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

AI总结 RADIO通过推理提取和基于推理的对齐方法，弥合检索增强生成中重排器与生成器之间的相关性差距。

Comments Accepted to ACL 25 Findings

2512.06874 2025-12-09 cs.CL 57%

An Analysis of Large Language Models for Simulating User Responses in Surveys

大型语言模型在模拟调查用户响应中的分析

Ziyun Yu, Yiru Zhou, Chen Zhao, Hongyi Wen

机构 * Center for Data Science, NYU Shanghai（纽约大学上海分校数据科学中心）； New York University（纽约大学）

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL

AI总结本文分析了大型语言模型在模拟调查用户响应中的表现，提出CLAIMSIM方法以提高响应多样性，但发现模型在处理不同人口特征时存在固有偏见和推理限制。

Comments Accepted to IJCNLP-AACL 2025 (Main Conference)

2512.06266 2025-12-09 cs.CL 57%

Nanbeige4-3B Technical Report: Exploring the Frontier of Small Language Models

Nanbeige4-3B 技术报告：探索小型语言模型的前沿

Chen Yang, Guangyue Peng, Jiaying Zhu, Ran Le, Ruixiang Feng, Tao Zhang, Wei Ruan, Xiaoqi Liu, Xiaoxue Cheng, Xiyun Xu, Yang Song, Yanzipeng Gao, Yiming Jia, Yun Xing, Yuntao Wen, Zekai Wang, Zhenwei An, Zhicong Sun, Zongchao Chen

机构 * Nanbeige LLM Lab（纳布吉LLM实验室）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

AI总结 Nanbeige4-3B通过FG-WSD调度器、DPD蒸馏和强化学习技术，实现了小型语言模型在性能和扩展定律上的突破。

2512.06020 2025-12-09 cs.CV cs.AI 57%

PrefGen: Multimodal Preference Learning for Preference-Conditioned Image Generation

PrefGen: 多模态偏好学习用于偏好条件下的图像生成

Wenyi Mo, Tianyu Zhang, Yalong Bai, Ligong Han, Ying Ba, Dimitris N. Metaxas

机构 * Rutgers University（罗格斯大学）； iN2X ； MIT-IBM Watson AI Lab（麻省理工-IBM沃森人工智能实验室）； Red Hat AI Innovation（红帽人工智能创新）

专题命中偏好对齐：alignment(abstract)；分类 cs.AI

AI总结 PrefGen通过多模态大语言模型提取用户偏好并注入扩散模型，实现个性化图像生成，优于现有方法。

Comments Project Page: \href{https://prefgen.github.io/}{\texttt{https://prefgen.github.io}}

2511.16139 2025-12-05 cs.AI 57%

Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints

通过几何投影参考约束的多维评分导向奖励模型学习

Yongnan Jin, Xurui Li, Feng Cao, Liucun Gao, Juanjuan Yao

机构 * Shanghai Mingpin Medical Data Technology Co., Ltd.（上海明品医疗数据技术有限公司）

专题命中偏好对齐：alignment(abstract)；分类 cs.AI

AI总结通过几何投影参考约束的多维评分导向奖励模型学习，提升医疗领域大型语言模型的性能和对齐能力。

2512.04419 2025-12-05 cs.AI 57%

Solving LLM Repetition Problem in Production: A Comprehensive Study of Multiple Solutions

在生产环境中解决大语言模型重复问题：对多种解决方案的综合研究

Weiwei Wang, Weijie Zou, Jiyong Min

机构 * Shenzhen Sunline Tech Co., Ltd（深圳Sunline科技有限公司）

专题命中偏好对齐：DPO(abstract)；分类 cs.AI

AI总结本文针对大语言模型在生产环境中重复问题，提出多种解决方案并验证其有效性，通过理论分析和实验评估，识别关键参数并提供实用的生产级方法。

2512.04302 2025-12-05 cs.AI 57%

Towards better dense rewards in Reinforcement Learning Applications

迈向强化学习应用中更优质的密集奖励

Shuyuan Zhang

机构 * McGill University（麦吉尔大学）； Mila

专题命中偏好对齐：alignment(abstract)；分类 cs.AI

AI总结本文探讨了强化学习应用中如何构建更有效的密集奖励，通过逆强化学习、奖励建模和自监督学习等方法提高奖励的准确性和可靠性。

Comments arXiv admin note: substantial text overlap with arXiv:2505.20417

2503.18929 2025-12-04 cs.LG 57%

Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training

轨迹平衡与异步性：解耦探索与学习以实现快速、可扩展的LLM后训练

Brian Bartoldson, Siddarth Venkatraman, James Diffenderfer, Moksh Jain, Tal Ben-Nun, Seanie Lee, Minsu Kim, Johan Obando-Ceron, Yoshua Bengio, Bhavya Kailkhura

机构 * Lawrence Livermore National Laboratory（劳伦斯利弗莫尔国家实验室）； Mila – Quebec AI Institute（魁北克AI研究院）； Université de Montréal（蒙特利尔大学）； KAIST（韩国科学技术院）； CIFAR Fellow

专题命中偏好对齐：DPO(abstract)；分类 cs.LG

AI总结 TBA通过解耦探索与学习，提升LLM后训练的速度和性能，适用于多种任务并支持大规模数据生成。

Comments NeurIPS 2025; 27 pages

2512.03001 2025-12-03 cs.AI 57%

Invasive Context Engineering to Control Large Language Models

侵入式上下文工程以控制大语言模型

Thomas Rivasseau

机构 * McGill University（麦吉尔大学）

专题命中偏好对齐：jailbreak(abstract)；分类 cs.AI

AI总结本文提出侵入式上下文工程方法，通过在LLM上下文中插入控制句子以提升其安全性，避免长上下文场景下的数据短缺问题。

Comments 4 pages

2510.20358 2025-12-02 cs.CL 57%

Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)

对话不足以造就一个交际性的婴儿语言模型（但也不如发展启发式强化学习）

Francesca Padovani, Bastian Bunzeck, Manar Ali, Omar Momen, Arianna Bisazza, Hendrik Buschmeier, Sina Zarrieß

机构 * Center for Language and Cognition (CLCG), University of Groningen（语言与认知中心（CLCG）、格罗宁根大学）； CRC 1646 – Linguistic Creativity in Communication, Bielefeld University（语言交流创造性研究（CRC 1646）、比尔特诺夫大学）

专题命中偏好对齐：DPO(abstract)；分类 cs.CL

AI总结本文探讨了仅通过对话数据预训练是否能生成有效的小型语言模型，并通过多种微调策略提升模型的交际能力，发现DPO微调在自定义对话基准测试中表现更优。

Journal ref Proceedings of the First BabyLM Workshop (2025), pp. 421-435

2507.18624 2025-12-02 cs.CL 57%

Checklists Are Better Than Reward Models For Aligning Language Models

检查表比奖励模型更能对齐语言模型

Vijay Viswanathan, Yanchao Sun, Shuang Ma, Xiang Kong, Meng Cao, Graham Neubig, Tongshuang Wu

机构 * Carnegie Mellon University（卡内基梅隆大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

AI总结本文提出基于检查表反馈的强化学习方法，通过灵活的指令特定准则提升语言模型的指令遵循能力，在多个基准测试中均取得性能提升。

Comments Presented at NeurIPS 2025

2511.21638 2025-11-27 cs.LG 57%

Aligning LLMs Toward Multi-Turn Conversational Outcomes Using Iterative PPO

通过迭代PPO对齐大语言模型以实现多轮对话结果

Daniel R. Jiang, Jalaj Bhandari, Yukai Yang, Rémi Munos, Tyler Lu

机构 * Meta ； FAIR at Meta（Meta的FAIR）

专题命中偏好对齐：RLHF(abstract)；分类 cs.LG

AI总结本文提出Iterative PPO方法，通过迭代优化多轮对话中的Q函数和策略，实现更稳定的对话生成。

Comments 12 pages, 2 figures

2511.17869 2025-11-25 cs.LG 57%

The Horcrux: Mechanistically Interpretable Task Decomposition for Detecting and Mitigating Reward Hacking in Embodied AI Systems

霍克鲁斯：用于检测和缓解具身AI系统中奖励黑客行为的可解释任务分解

Subramanyam Sahoo, Jared Junkin

机构 * Berkeley AI Safety Initiative (BASIS) UC Berkeley（伯克利人工智能安全计划（BASIS）伯克利大学）； Department of Electrical and Computer Engineering Johns Hopkins University（电气与计算机工程系约翰霍普金斯大学）

专题命中偏好对齐：RLHF(abstract)；分类 cs.LG

AI总结本研究提出MITD方法，通过可解释性任务分解有效检测和缓解具身AI系统中的奖励黑客行为，实验表明分解深度可显著降低奖励黑客频率。

Comments Accepted to the NeurIPS (Mexico City) 2025 Workshop on Embodied and Safe-Assured Robotic Systems (E-SARS). Thanks to Aman Chadha

2409.19753 2025-11-21 cs.CL 57%

CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering

CoTKR: 基于链式推理的增强型知识重写用于复杂知识图谱问答

Yike Wu, Yi Huang, Nan Hu, Yuncheng Hua, Guilin Qi, Jiaoyan Chen, Jeff Z. Pan

机构 * Southeast University（东南大学）； Key Laboratory of New Generation Artificial Intelligence Technology and Its Interdisciplinary Applications (Southeast University) Ministry of Education（新一代人工智能技术及其交叉应用国家重点实验室）； China Mobile Research Institute（中国移动研究院）； Monash University（墨尔本大学）； University of Manchester（曼彻斯特大学）； University of Edinburgh（爱丁堡大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

AI总结 CoTKR通过链式推理增强知识重写方法，提升复杂知识图谱问答性能。

2511.15573 2025-11-20 cs.CY 57%

Two-Faced Social Agents: Context Collapse in Role-Conditioned Large Language Models

Vikram K Suresh

专题命中偏好对齐：alignment(abstract)；分类 cs.CY

2511.15038 2025-11-20 cs.SD cs.AI eess.AS 57%

Aligning Generative Music AI with Human Preferences: Methods and Challenges

Dorien Herremans, Abhinaba Roy

专题命中偏好对齐：alignment(abstract)；分类 cs.AI

Comments Accepted at the AAAI-2026 Senior Member Track

2511.13689 2025-11-19 cs.CL cs.CV 57%

Crossing Borders: A Multimodal Challenge for Indian Poetry Translation and Image Generation

Sofia Jamil, Kotla Sai Charan, Sriparna Saha, Koustava Goswami, Joseph K J

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

2511.12464 2025-11-18 cs.CL 57%

Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models

Chenglong Wang, Yifu Huo, Yang Gan, Yongyu Mu, Qiaozhi He, Murun Yang, Bei Li, Chunliang Zhang, Tongran Liu, Anxiang Ma, Zhengtao Yu, Jingbo Zhu, Tong Xiao

机构 * School of Computer Science and Engineering, Northeastern University（东北大学计算机科学与工程学院）； Meituan Inc.（美团公司）； NiuTrans Research（牛译研所）； CAS Key Laboratory of Behavioral Science, Institute of Psychology, CAS（中国科学院行为科学重点实验室）； Kunming University of Science and Technology（昆明理工大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments Accepted by AAAI 2026

2503.10071 2025-11-18 cs.AI 57%

Advanced Tool Learning and Selection System (ATLASS): A Closed-Loop Framework Using LLM

Mohd Ariful Haque, Justin Williams, Sunzida Siddique, Md. Hujaifa Islam, Hasmot Ali, Kishor Datta Gupta, Roy George

机构 * Clark Atlanta University（克拉克亚特兰大大学）； Daffodil International University（达福尔国际大学）； Ahsanullah University of Science and Technology（阿沙努拉大学科学与技术学院）

专题命中偏好对齐：safety(abstract)；分类 cs.AI

Journal ref 2025 IEEE International Conference on Service-Oriented System Engineering (SOSE), Tucson, AZ, USA, 21-24 July 2025, IEEE

2508.18212 2025-11-18 cs.CL 57%

Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries

Meiling Ning, Zhongbao Zhang, Junda Ye, Jiabao Guo, Qingyuan Guan

专题命中偏好对齐：RLHF(abstract)；分类 cs.CL

Comments After further internal discussion, our author team has decided to withdraw this submission due to the need for several important refinements to the manuscript. All co-authors have been informed and agree with this decision

2508.02087 2025-11-13 cs.CL 57%

When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Models

Keyu Wang, Jin Li, Shu Yang, Zhuoran Zhang, Di Wang

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

2508.20916 2025-11-11 cs.CL 57%

SageLM: A Multi-aspect and Explainable Large Language Model for Speech Judgement

Yuan Ge, Junxiang Zhang, Xiaoqian Liu, Bei Li, Xiangnan Ma, Chenglong Wang, Kaiyang Ye, Yangfan Du, Linfeng Zhang, Yuxin Huang, Tong Xiao, Zhengtao Yu, JingBo Zhu

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

2505.09039 2025-11-11 cs.CL 57%

Atomic Consistency Preference Optimization for Long-Form Question Answering

Jingfeng Chen, Raghuveer Thirukovalluru, Junlin Wang, Kaiwei Luo, Bhuwan Dhingra

专题命中偏好对齐：alignment(abstract)；分类 cs.CL

Comments 13 pages, 1 figure

2511.04845 2025-11-10 cs.LG 57%

Investigating U.S. Consumer Demand for Food Products with Innovative Transportation Certificates Based on Stated Preferences and Machine Learning Approaches

Jingchen Bi, Rodrigo Mesa-Arango

专题命中偏好对齐：safety(abstract)；分类 cs.LG

2410.03740 2025-11-10 cs.CL 57%

LEME: Open Large Language Models for Ophthalmology with Advanced Reasoning and Clinical Validation

Hyunjae Kim, Xuguang Ai, Sahana Srinivasan, Aidan Gilson, Maxwell B. Singer, Krithi Pushpanathan, Qianqian Xie, Jungwoo Park, Serina Applebaum, Gabriel Dawei Yang, Minjie Zou, David Ziyou Chen, Ke Zou, Soshian Sarrafpour, Ji Liu, Yu Yin, Jimin Huang, Quang Ngoc Nguyen, Erping Long, Peixing Wan, Dianbo Liu, Richard Hintz, W. Jim Zheng, Sophia Y. Wang, Lucila Ohno-Machado, Hua Xu, Ron A. Adelman, Luciano V. Del Priore, Yih-Chung Tham, Qingyu Chen

专题命中偏好对齐：safety(abstract)；分类 cs.CL

2506.07127 2025-10-31 cs.RO cs.AI 57%

Human-assisted Robotic Policy Refinement via Action Preference Optimization

Wenke Xia, Yichu Yang, Hongtao Wu, Xiao Ma, Tao Kong, Di Hu

机构 * Gaoling School of Artificial Intelligence, Renmin University of China, Beijing（中国人民大学北京校区人工智能学院）； Engineering Research Center of Next-Generation Intelligent Search（下一代智能搜索与推荐工程研究中心）； Beijing Key Laboratory of Research on Large Models（北京大型模型研究重点实验室）

专题命中偏好对齐：alignment(abstract)；分类 cs.AI

Comments Accepted By NeurIPS 2025

2510.21143 2025-10-29 cs.AI 57%

PanicToCalm: A Proactive Counseling Agent for Panic Attacks

Jihyun Lee, Yejin Min, San Kim, Yejin Jeon, SungJun Yang, Hyounghun Kim, Gary Geunbae Lee

机构 * Graduate School of Artificial Intelligence, POSTECH（人工智能研究生院，POSTECH）； Department of Computer Science and Engineering, POSTECH（计算机科学与工程系，POSTECH）

专题命中偏好对齐：alignment(abstract)；分类 cs.AI

Comments Accepted in EMNLP 2025

2505.22092 2025-10-29 cs.AI 57%

VIRAL: Vision-grounded Integration for Reward design And Learning

Valentin Cuzin-Rambaud, Emilien Komlenovic, Alexandre Faure, Bruno Yun

机构 * Université Claude Bernard Lyon 1（克莱尔伯恩大学里昂1分校）； CNRS（国家科学研究中心）； Ecole Centrale de Lyon（里昂中央理工学院）； INSA Lyon（里昂工业高等学院）； Université Lumière Lyon 2（里昂2大学卢米埃尔分校）； LIRIS（图像研究所）

专题命中偏好对齐：alignment(abstract)；分类 cs.AI

2510.21847 2025-10-28 cs.LG 57%

SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization

Kaiyi Xu, Junchao Gong, Wenlong Zhang, Ben Fei, Lei Bai, Wanli Ouyang

机构 * University of Science and Technology of China（中国科学技术大学）； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）； Shanghai Jiao Tong University（上海交通大学）； Chinese University of Hong Kong（香港中文大学）

专题命中偏好对齐：alignment(abstract)；分类 cs.LG

1. 偏好对齐 3243 篇

QiMeng-SALV: Signal-Aware Learning for Verilog Code Generation

Bridging Relevance and Reasoning: Rationale Distillation in Retrieval-Augmented Generation

An Analysis of Large Language Models for Simulating User Responses in Surveys

Nanbeige4-3B Technical Report: Exploring the Frontier of Small Language Models

PrefGen: Multimodal Preference Learning for Preference-Conditioned Image Generation

Multidimensional Rubric-oriented Reward Model Learning via Geometric Projection Reference Constraints

Solving LLM Repetition Problem in Production: A Comprehensive Study of Multiple Solutions

Towards better dense rewards in Reinforcement Learning Applications

Trajectory Balance with Asynchrony: Decoupling Exploration and Learning for Fast, Scalable LLM Post-Training

Invasive Context Engineering to Control Large Language Models

Dialogue Is Not Enough to Make a Communicative BabyLM (But Neither Is Developmentally Inspired Reinforcement Learning)

Checklists Are Better Than Reward Models For Aligning Language Models

Aligning LLMs Toward Multi-Turn Conversational Outcomes Using Iterative PPO

The Horcrux: Mechanistically Interpretable Task Decomposition for Detecting and Mitigating Reward Hacking in Embodied AI Systems

CoTKR: Chain-of-Thought Enhanced Knowledge Rewriting for Complex Knowledge Graph Question Answering

Two-Faced Social Agents: Context Collapse in Role-Conditioned Large Language Models

Aligning Generative Music AI with Human Preferences: Methods and Challenges

Crossing Borders: A Multimodal Challenge for Indian Poetry Translation and Image Generation

Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models

Advanced Tool Learning and Selection System (ATLASS): A Closed-Loop Framework Using LLM

Better Language Model-Based Judging Reward Modeling through Scaling Comprehension Boundaries

When Truth Is Overridden: Uncovering the Internal Origins of Sycophancy in Large Language Models

SageLM: A Multi-aspect and Explainable Large Language Model for Speech Judgement

Atomic Consistency Preference Optimization for Long-Form Question Answering

Investigating U.S. Consumer Demand for Food Products with Innovative Transportation Certificates Based on Stated Preferences and Machine Learning Approaches

LEME: Open Large Language Models for Ophthalmology with Advanced Reasoning and Clinical Validation

Human-assisted Robotic Policy Refinement via Action Preference Optimization

PanicToCalm: A Proactive Counseling Agent for Panic Attacks

VIRAL: Vision-grounded Integration for Reward design And Learning

SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization