大模型对齐与安全 - arXivDaily 专题

2305.19223 2023-05-31 cs.AI cs.CY cs.HC 87%

Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety

Catalin Mitelut, Ben Smith, Peter Vamplew

专题命中安全训练：safety(title)；AI safety(title)；alignment(abstract)；分类 cs.AI、cs.CY

1606.06565 2016-07-26 cs.AI cs.LG 87%

Concrete Problems in AI Safety

Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman, Dan Mané

专题命中安全训练：safety(title,abstract)；AI safety(title)；分类 cs.AI、cs.LG

Comments 29 pages

2607.02781 2026-07-07 cs.LG cs.AI cs.CL 新提交 87%

Safe Inference-Time Alignment via Lagrangian Reward Augmentation

通过拉格朗日奖励增强实现安全推理时对齐

Yaswanth Chittepu, Ativ Joshi, Sohini Chintala, Scott Niekum

机构 * University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）； Independent Researcher（独立研究者）

专题命中安全训练：alignment(title,abstract)；safety(abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结研究提出拉格朗日奖励增强框架，从含奖励与成本模型的约束目标出发，经对偶化将问题转化为一维凸问题，校准对偶变量获增强奖励，提升推理时对齐的有益性与无害性权衡。

2606.28347 2026-06-30 cs.CY cs.AI cs.LG 87%

Agentic Safety is an Epistemic Property, Not a Behavioral One

智能体安全是认知属性，而非行为属性

Charles L. Wang, Keir Dorchen, Peter Jin

专题命中安全训练：safety(title,abstract)；alignment(abstract)；AI safety(abstract)；分类 cs.AI、cs.CY、cs.LG

AI总结本文提出AI安全应视为认知属性，强调系统需保持可教性（teachability），即在学习、适应和自修改过程中仍能被纠正。

Comments To appear in proceedings of ICML 2026

2603.15417 2026-03-17 cs.LG cs.AI cs.CL cs.CR 87%

Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities

测试时强化学习中的放大效应：安全性和推理漏洞

Vanshaj Khattar, Md Rafi ur Rashid, Moumita Choudhury, Jing Liu, Toshiaki Koike-Akino, Ming Jin, Ye Wang

机构 * Penn State University（宾夕法尼亚州立大学）； University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校）； Mitsubishi Electric Research Laboratories（三菱电机研究实验室）

专题命中安全训练：safety(title,abstract)；jailbreak(abstract)；prompt injection(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结本文研究了测试时训练方法的安全性漏洞，发现测试时强化学习中有害提示注入会放大模型行为，导致推理能力下降。

2507.01915 2025-07-03 cs.CL cs.AI cs.LG 87%

Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models

Chengao Li, Hanyu Zhang, Yunkun Xu, Hongyan Xue, Xiang Ao, Qing He

机构 * Key Lab of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (CAS)（智能信息处理重点实验室，计算技术研究所，中国科学院）； State Key Lab of AI Safety（人工智能安全国家重点实验室）； University of Chinese Academy of Sciences, CAS（中国科学院大学）； Zhejiang University（浙江大学）

专题命中安全训练：alignment(title,abstract)；RLHF(abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments 19 pages, 3 figures. Accepted by ACL 2025 (main)

2509.22510 2026-05-19 cs.CL 86%

We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrong

我们思考，因此我们对LLMs进行对齐，使其在出错前变得有益、无害和诚实

Gautam Siddharth Kashyap, Mark Dras, Usman Naseem

机构 * School of Computing Macquarie University（计算机学院麦Quarie大学）

专题命中安全训练：RLHF(abstract,abstract_cn)；harmlessness(abstract,abstract_cn)；alignment(abstract)；trustworthy(abstract)

AI总结本文提出AMBS框架，通过在1-to-N Transformer设置中参数化目标特定转换，解决LLM对齐中多个目标之间的干扰问题，提升HHH目标的联合满足能力。

2509.22745 2025-10-10 cs.CR cs.AI 86%

Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignment

Jaehan Kim, Minkyoo Song, Seungwon Shin, Sooel Son

机构 * KAIST（韩国科学技术院）

专题命中安全训练：safety(title,abstract)；alignment(title)；分类 cs.AI

Comments Under review

2509.25457 2025-10-01 cs.HC cs.CY 86%

Human vs. AI Safety Perception? Decoding Human Safety Perception with Eye-Tracking Systems, Street View Images, and Explainable AI

Yuhao Kang, Junda Chen, Liu Liu, Kshitij Sharmad, Martina Mazzarello, Simone Mora, Fabio Duarte, Carlo Ratti

专题命中安全训练：safety(title,abstract)；AI safety(title)；分类 cs.CY

Comments 28 pages, 8 figures

2501.06208 2025-08-13 cs.CL 86%

Enhancing AI Safety Through the Fusion of Low Rank Adapters

Satya Swaroop Gudipudi, Sreeram Vipparla, Harpreet Singh, Shashwat Goel, Ponnurangam Kumaraguru

机构 * IIIT Hyderabad（海德拉巴国家理工学院）； NSUT Delhi（德里NSUT）

专题命中安全训练：safety(title,abstract)；AI safety(title)；分类 cs.CL

2502.08142 2025-02-13 cs.AI 86%

Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy LLM Inferences

Shanshan Han, Salman Avestimehr, Chaoyang He

专题命中安全训练：safety(title,abstract)；trustworthy(title)；分类 cs.AI

Comments arXiv admin note: text overlap with arXiv:2406.10847

1906.03466 2019-06-11 cs.AI cs.CR cs.CV 86%

Strategies to architect AI Safety: Defense to guard AI from Adversaries

Rajagopal. A, Nirmala. V

专题命中安全训练：safety(title,abstract)；AI safety(title)；分类 cs.AI

1805.00899 2018-10-23 stat.ML cs.LG 86%

AI safety via debate

Geoffrey Irving, Paul Christiano, Dario Amodei

专题命中安全训练：safety(title)；AI safety(title)；alignment(abstract)；分类 cs.LG

Comments 24 pages, 6 figures

2606.31876 2026-07-07 cs.AI cs.CV cs.LG 新提交 86%

Harnessing Textual Refusal Directions for Multimodal Safety

利用文本拒绝方向实现多模态安全

Moreno D'Incà, Nicu Sebe, Massimiliano Mancini

机构 * University of Trento（特伦托大学）

专题命中安全训练：safety(title,abstract)；alignment(abstract)；jailbreak(abstract)；分类 cs.AI、cs.LG

AI总结提出MARS方法，通过从LLM骨干提取的文本拒绝方向泛化到多模态，无需多模态安全数据即可提升安全性，在五个先进MLLM上验证了有效性。

Comments Preprint

2606.17872 2026-06-17 cs.LG cs.AI 新提交 86%

AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor

AnchorKV: 通过拒绝锚点的软惩罚实现安全感知的KV缓存压缩

Ning Ni, Yingjie Lao

机构 * Department of Computer Science, Tufts University（塔夫茨大学计算机科学系）； Department of Electrical and Computer Engineering, Tufts University（塔夫茨大学电气与计算机工程系）

专题命中安全训练：safety(title,abstract)；alignment(abstract)；jailbreak(abstract)；分类 cs.AI、cs.LG

AI总结提出AnchorKV，一种通过软惩罚机制调整令牌保留分数以远离有害提示的KV缓存压缩方法，在保持实用性的同时显著提升安全性。

2604.26516 2026-04-30 cs.LG cs.AI 86%

Lyapunov-Guided Self-Alignment: Test-Time Adaptation for Offline Safe Reinforcement Learning

Lyapunov引导的自对齐：用于离线安全强化学习的测试时适应

Seungyub Han, Hyungjin Kim, Jungwoo Lee

机构 * Seoul National University（首尔国立大学）

专题命中安全训练：alignment(title,abstract)；safety(abstract,abstract_cn)；分类 cs.AI、cs.LG

AI总结本文提出SAS框架，通过自对齐机制在不重新训练的情况下实现离线安全强化学习的测试时适应，利用Lyapunov条件筛选可行轨迹并生成上下文提示，提升安全性与性能。

Comments Accepted at AISTATS 2026. First two authors contributed equally. Project page: https://seungyubhan.github.io/sas/. Code: https://github.com/seungyubhan/sas

2507.20704 2026-03-05 cs.CL cs.AI cs.CR 86%

Text2VLM: Adapting Text-Only Datasets to Evaluate Alignment Training in Visual Language Models

Text2VLM: 将文本-only数据集适应于评估视觉语言模型对齐训练

Gabriel Downer, Sean Craven, Damian Ruck, Jake Thomas

机构 * Advai

专题命中安全训练：alignment(title,abstract)；safety(abstract)；prompt injection(abstract)；分类 cs.CL、cs.AI

AI总结 Text2VLM通过将文本-only数据集转换为多模态格式，评估VLMs对提示注入攻击的鲁棒性，并揭示模型对齐中的关键弱点。

Comments 9 pages, 9 figures. Jake Thomas served as Editor for this manuscript

Journal ref Proceedings of the 2025 Conference on Applied Machine Learning for Information Security, Proceedings of Machine Learning Research PMLR Vol 299 pp 28 41

2601.18730 2026-01-27 cs.CL cs.LG 86%

Reflect: Transparent Principle-Guided Reasoning for Constitutional Alignment at Scale

Reflect: 为大规模宪法对齐的透明原则引导推理

Henry Bell, Caroline Zhang, Mohammed Mobasserul Haque, Dhaval Potdar, Samia Zaman, Brandon Fain

机构 * Duke University（杜克大学）； Independent Researcher（独立研究者）

专题命中安全训练：alignment(title,abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.LG

AI总结 Reflect通过透明推理框架在不需训练数据的情况下提升大语言模型对多样原则的对齐能力，增强安全性和鲁棒性。

2507.09709 2026-01-22 cs.CL cs.LG 86%

Large Language Models Encode Semantics and Alignment in Linearly Separable Representations

大语言模型在线性可分的表示中编码语义和对齐

Baturay Saglam, Paul Kassianik, Blaine Nelson, Sajana Weerawardhena, Yaron Singer, Amin Karbasi

机构 * Yale University（耶鲁大学）； Foundation AI – Cisco Systems Inc（Foundation AI – 卡西欧系统公司）

专题命中安全训练：alignment(title,abstract)；safety(abstract)；prompt injection(abstract)；分类 cs.CL、cs.LG

AI总结本研究发现大语言模型通过线性可分的表示编码语义和对齐，提出基于潜在空间的MLP探测器有效提升安全防护。

Comments IJCNLP and the Asian Chapter of ACL

2601.10141 2026-01-16 cs.LG cs.AI 86%

Understanding and Preserving Safety in Fine-Tuned LLMs

理解并保持在微调大语言模型中的安全性

Jiawen Zhang, Yangfan Hu, Kejia Chen, Lipeng He, Jiachen Ma, Jian Lou, Dan Li, Jian Liu, Xiaohu Yang, Ruoxi Jia

机构 * Zhejiang University（浙江大学）； University of Wisconsin–Madison（威斯康星大学麦迪逊分校）； University of Waterloo（滑铁卢大学）； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室）； Sun Yat-sen University（中山大学）

专题命中安全训练：safety(title,abstract)；alignment(abstract)；jailbreak(abstract)；分类 cs.AI、cs.LG

AI总结本文提出安全保持微调（SPF）方法，通过分析安全性与实用性梯度的几何关系，有效解决微调过程中安全性与实用性之间的矛盾，保持模型性能并恢复预训练的安全性对齐。

2601.01836 2026-01-06 cs.AI cs.CY 86%

COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs

COMPASS：一个评估大语言模型在组织特定政策对齐性的框架

Dasol Choi, DongGeon Lee, Brigitta Jesica Kartono, Helena Berndt, Taeyoun Kwon, Joonwon Jang, Haon Park, Hwanjo Yu, Minsuk Kahng

机构 * AIM Intelligence（AIM智能科技）； BMW Group（宝马集团）； Yonsei University（延世大学）； POSTECH ； Seoul National University（首尔国立大学）

专题命中安全训练：alignment(title,abstract)；safety(abstract)；AI safety(abstract)；分类 cs.AI、cs.CY

AI总结 COMPASS框架通过评估大语言模型在组织特定政策下的对齐性，揭示了模型在合规性与对抗性鲁棒性上的显著差异，为组织AI安全提供了关键评估工具。

2505.14667 2025-10-24 cs.AI cs.CL 86%

SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment

Wonje Jeung, Sangyeon Yoon, Minsuk Kahng, Albert No

机构 * Department of Artificial Intelligence, Yonsei University（人工智能系，延世大学）； Department of Computer Science and Engineering, Yonsei University（计算机科学与工程系，延世大学）

专题命中安全训练：alignment(title,abstract)；safety(abstract)；jailbreak(abstract)；分类 cs.CL、cs.AI

Comments Accepted at NeurIPS 2025. Code and models are available at https://ai-isl.github.io/safepath

2510.02194 2025-10-03 cs.AI cs.CR cs.LG 86%

UpSafe$^\circ$C: Upcycling for Controllable Safety in Large Language Models

Yuhao Sun, Zhuoer Xu, Shiwen Cui, Kun Yang, Lingyun Yu, Yongdong Zhang, Hongtao Xie

专题命中安全训练：safety(title,abstract)；alignment(abstract)；jailbreak(abstract)；分类 cs.AI、cs.LG

2502.01208 2025-06-23 cs.LG cs.CL 86%

On Almost Surely Safe Alignment of Large Language Models at Inference-Time

Xiaotong Ji, Shyam Sundhar Ramesh, Matthieu Zimmer, Ilija Bogunovic, Jun Wang, Haitham Bou Ammar

机构 * Huawei Noah’s Ark Lab（华为诺亚实验室）； Imperial College London（帝国理工学院伦敦分校）； University College London（伦敦大学学院）； UCL Centre for Artificial Intelligence（UCL人工智能中心）

专题命中安全训练：alignment(title,abstract)；RLHF(abstract)；safety(abstract)；分类 cs.CL、cs.LG

2405.00557 2025-06-03 cs.CL cs.AI 86%

Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment

Zhili Liu, Yunhao Gou, Kai Chen, Lanqing Hong, Jiahui Gao, Fei Mi, Yu Zhang, Zhenguo Li, Xin Jiang, Qun Liu, James T. Kwok

机构 * The Hong Kong University of Science and Technology（香港科学与技术大学）； Huawei Noah’s Ark Lab（华为诺亚实验室）； Southern University of Science and Technology（南方科技大学）

专题命中安全训练：alignment(title,abstract)；safety(abstract)；jailbreak(abstract)；分类 cs.CL、cs.AI

2401.16332 2025-05-28 cs.CL cs.AI 86%

Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methods

Yotam Wolf, Noam Wies, Dorin Shteyman, Binyamin Rothberg, Yoav Levine, Amnon Shashua

机构 * The Hebrew University（希伯来大学）

专题命中安全训练：alignment(title,abstract)；safety(abstract)；AI safety(abstract)；分类 cs.CL、cs.AI

2504.10081 2025-04-15 cs.AI cs.CL 86%

RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability

Yichi Zhang, Zihao Zeng, Dongbai Li, Yao Huang, Zhijie Deng, Yinpeng Dong

专题命中安全训练：safety(title,abstract)；alignment(abstract)；jailbreak(abstract)；分类 cs.CL、cs.AI

2503.17882 2025-03-25 cs.CL cs.AI 86%

Think Before Refusal : Triggering Safety Reflection in LLMs to Mitigate False Refusal Behavior

Shengyun Si, Xinpeng Wang, Guangyao Zhai, Nassir Navab, Barbara Plank

专题命中安全训练：safety(title,abstract)；alignment(abstract)；harmlessness(abstract)；分类 cs.CL、cs.AI

Comments 18 pages, 23 figures

2410.06625 2025-02-11 cs.CV cs.CL cs.LG 86%

ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time

Yi Ding, Bolian Li, Ruqi Zhang

专题命中安全训练：safety(title,abstract)；alignment(abstract)；harmlessness(abstract)；分类 cs.CL、cs.LG

Comments 29pages

2408.07663 2024-12-20 cs.CL cs.AI 86%

Alignment-Enhanced Decoding:Defending via Token-Level Adaptive Refining of Probability Distributions

Quan Liu, Zhenhong Zhou, Longzhu He, Yi Liu, Wei Zhang, Sen Su

专题命中安全训练：alignment(title,abstract)；safety(abstract)；jailbreak(abstract)；分类 cs.CL、cs.AI

Comments Accepted by EMNLP 2024, 15 pages, 5 figures

1. 安全训练 3242 篇

Intent-aligned AI systems deplete human agency: the need for agency foundations research in AI safety

Concrete Problems in AI Safety

Safe Inference-Time Alignment via Lagrangian Reward Augmentation

Agentic Safety is an Epistemic Property, Not a Behavioral One

Amplification Effects in Test-Time Reinforcement Learning: Safety and Reasoning Vulnerabilities

Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Models

We Think, Therefore We Align LLMs to Helpful, Harmless and Honest Before They Go Wrong

Defending MoE LLMs against Harmful Fine-Tuning via Safety Routing Alignment

Human vs. AI Safety Perception? Decoding Human Safety Perception with Eye-Tracking Systems, Street View Images, and Explainable AI

Enhancing AI Safety Through the Fusion of Low Rank Adapters

Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy LLM Inferences

Strategies to architect AI Safety: Defense to guard AI from Adversaries

AI safety via debate

Harnessing Textual Refusal Directions for Multimodal Safety

AnchorKV: Safety-Aware KV Cache Compression via Soft Penalty with a Refusal Anchor

Lyapunov-Guided Self-Alignment: Test-Time Adaptation for Offline Safe Reinforcement Learning

Text2VLM: Adapting Text-Only Datasets to Evaluate Alignment Training in Visual Language Models

Reflect: Transparent Principle-Guided Reasoning for Constitutional Alignment at Scale

Large Language Models Encode Semantics and Alignment in Linearly Separable Representations

Understanding and Preserving Safety in Fine-Tuned LLMs

COMPASS: A Framework for Evaluating Organization-Specific Policy Alignment in LLMs

SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment

UpSafe$^\circ$C: Upcycling for Controllable Safety in Large Language Models

On Almost Surely Safe Alignment of Large Language Models at Inference-Time

Mixture of insighTful Experts (MoTE): The Synergy of Thought Chains and Expert Mixtures in Self-Alignment

Tradeoffs Between Alignment and Helpfulness in Language Models with Steering Methods

RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability

Think Before Refusal : Triggering Safety Reflection in LLMs to Mitigate False Refusal Behavior

ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time

Alignment-Enhanced Decoding:Defending via Token-Level Adaptive Refining of Probability Distributions