大模型对齐与安全

Integrative Biological Simulation, Neuropsychology, and AI Safety

Gopal P. Sarma, Adam Safron, Nick J. Hay

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；alignment(abstract)；分类 cs.AI、cs.LG

Comments 5 pages

Journal ref Proceedings of the AAAI Workshop on Artificial Intelligence Safety 2019 co-located with the Thirty-Third AAAI Conference on Artificial Intelligence 2019 (AAAI 2019)

AI Safety in Generative AI Large Language Models: A Survey

Jaymari Chua, Yun Li, Shiyi Yang, Chen Wang, Lina Yao

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；alignment(abstract)；分类 cs.CL、cs.CY

Scalable AI Safety via Doubly-Efficient Debate

Jonah Brown-Cohen, Geoffrey Irving, Georgios Piliouras

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；alignment(abstract)；分类 cs.AI、cs.LG

AI safety: state of the field through quantitative lens

Mislav Juric, Agneza Sandic, Mario Brcic

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；alignment(abstract)；分类 cs.AI、cs.CY

Comments 2020 43rd International Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO)

Foundational Challenges in Assuring Alignment and Safety of Large Language Models

Usman Anwar, Abulhair Saparov, Javier Rando, Daniel Paleka, Miles Turpin, Peter Hase, Ekdeep Singh Lubana, Erik Jenner, Stephen Casper, Oliver Sourbut, Benjamin L. Edelman, Zhaowei Zhang, Mario Günther, Anton Korinek, Jose Hernandez-Orallo, Lewis Hammond, Eric Bigelow, Alexander Pan, Lauro Langosco, Tomasz Korbak, Heidi Zhang, Ruiqi Zhong, Seán Ó hÉigeartaigh, Gabriel Recchia, Giulio Corsi, Alan Chan, Markus Anderljung, Lilian Edwards, Aleksandar Petrov, Christian Schroeder de Witt, Sumeet Ramesh Motwan, Yoshua Bengio, Danqi Chen, Philip H. S. Torr, Samuel Albanie, Tegan Maharaj, Jakob Foerster, Florian Tramer, He He, Atoosa Kasirzadeh, Yejin Choi, David Krueger

专题命中 其他安全 ：alignment(title,abstract)；safety(title,abstract)；分类 cs.CL、cs.AI、cs.CY

Mechanistic Interpretability for AI Safety -- A Review

Leonard Bereska, Efstratios Gavves

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；alignment(abstract)；分类 cs.AI

Comments Accepted to TMLR; for an HTML version, visit https://leonardbereska.github.io/blog/2024/mechinterpreview/

The Concept of Criticality in AI Safety

Yitzhak Spielberg, Amos Azaria

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；alignment(abstract)；分类 cs.AI

MESA: Improving MoE Safety Alignment via Decentralized Expertise

MESA: 通过去中心化专家提升MoE安全对齐

Yitong Sun, Yao Huang, Teng Li, Ranjie Duan, Yichi Zhang, Xingjun Ma, Hui Xue, Xingxing Wei

机构 * University of Science and Technology of China（中国科学技术大学） 纠错

专题命中 其他安全 ：alignment(title,abstract)；safety(title,abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 针对MoE架构中安全能力集中于少数专家导致的脆弱性，提出MESA框架，通过最优传输理论实现专家安全职责去中心化分配与路由细化，在保持实用性的同时提升防御性能。

Comments 18 pages, 8 figures, accepted by ICML 2026

CARE: Decoding Time Safety Alignment via Rollback and Introspection Intervention

Xiaomeng Hu, Fei Huang, Chenhan Yuan, Junyang Lin, Tsung-Yi Ho

机构 * Alibaba Group（阿里巴巴集团） ； The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 其他安全 ：alignment(title,abstract)；safety(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Re-Emergent Misalignment: How Narrow Fine-Tuning Erodes Safety Alignment in LLMs

Jeremiah Giordani

专题命中 其他安全 ：alignment(title,abstract)；safety(title,abstract)；分类 cs.CL、cs.AI、cs.LG

International AI Safety Report

Yoshua Bengio, Sören Mindermann, Daniel Privitera, Tamay Besiroglu, Rishi Bommasani, Stephen Casper, Yejin Choi, Philip Fox, Ben Garfinkel, Danielle Goldfarb, Hoda Heidari, Anson Ho, Sayash Kapoor, Leila Khalatbari, Shayne Longpre, Sam Manning, Vasilios Mavroudis, Mantas Mazeika, Julian Michael, Jessica Newman, Kwan Yee Ng, Chinasa T. Okolo, Deborah Raji, Girish Sastry, Elizabeth Seger, Theodora Skeadas, Tobin South, Emma Strubell, Florian Tramèr, Lucia Velasco, Nicole Wheeler, Daron Acemoglu, Olubayo Adekanmbi, David Dalrymple, Thomas G. Dietterich, Edward W. Felten, Pascale Fung, Pierre-Olivier Gourinchas, Fredrik Heintz, Geoffrey Hinton, Nick Jennings, Andreas Krause, Susan Leavy, Percy Liang, Teresa Ludermir, Vidushi Marda, Helen Margetts, John McDermid, Jane Munga, Arvind Narayanan, Alondra Nelson, Clara Neppel, Alice Oh, Gopal Ramchurn, Stuart Russell, Marietje Schaake, Bernhard Schölkopf, Dawn Song, Alvaro Soto, Lee Tiedrich, Gaël Varoquaux, Andrew Yao, Ya-Qin Zhang, Fahad Albalawi, Marwan Alserkal, Olubunmi Ajala, Guillaume Avrin, Christian Busch, André Carlos Ponce de Leon Ferreira de Carvalho, Bronwyn Fox, Amandeep Singh Gill, Ahmet Halit Hatip, Juha Heikkilä, Gill Jolly, Ziv Katzir, Hiroaki Kitano, Antonio Krüger, Chris Johnson, Saif M. Khan, Kyoung Mu Lee, Dominic Vincent Ligot, Oleksii Molchanovskyi, Andrea Monti, Nusu Mwamanzi, Mona Nemer, Nuria Oliver, José Ramón López Portillo, Balaraman Ravindran, Raquel Pezoa Rivera, Hammam Riza, Crystal Rugege, Ciarán Seoighe, Jerry Sheehan, Haroon Sheikh, Denise Wong, Yi Zeng

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；分类 cs.AI、cs.CY、cs.LG

Challenges in Ensuring AI Safety in DeepSeek-R1 Models: The Shortcomings of Reinforcement Learning Strategies

Manojkumar Parmar, Yuvaraj Govindarajulu

专题命中 其他安全 ：safety(title)；AI safety(title)；alignment(abstract)；harmlessness(abstract)

Comments 9 pages, 1 table

Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Models

Qin Liu, Chao Shang, Ling Liu, Nikolaos Pappas, Jie Ma, Neha Anna John, Srikanth Doss, Lluis Marquez, Miguel Ballesteros, Yassine Benajiba

专题命中 其他安全 ：alignment(title,abstract)；safety(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Preprint

Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Models

Fei Wang, Ninareh Mehrabi, Palash Goyal, Rahul Gupta, Kai-Wei Chang, Aram Galstyan

专题命中 其他安全 ：alignment(title,abstract)；safety(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted to EMNLP 2024 Main Conference. Project website: https://feiwang96.github.io/DataAdvisor/

Model Merging and Safety Alignment: One Bad Model Spoils the Bunch

Hasan Abed Al Kader Hammoud, Umberto Michieli, Fabio Pizzati, Philip Torr, Adel Bibi, Bernard Ghanem, Mete Ozay

专题命中 其他安全 ：alignment(title,abstract)；safety(title,abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Under review

How to Avoid Debate: Scalable AI Safety via Doubly-Efficient Interactive Proofs

如何避免辩论：通过双高效交互式证明实现可扩展的人工智能安全

Liyan Chen, Yael Tauman Kalai, Zoe Xi

机构 * MIT（麻省理工学院） 纠错

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；分类 cs.AI、cs.LG

AI总结 研究如何避免人工智能模型间的辩论来实现安全验证，提出单证明者交互式证明，给出双高效单证明者交互式证明及论证，用于神谕辅助计算，表明无辩论时交互式验证也可行。

Comments ICML 2026

Bridging Distribution Shift and AI Safety: Conceptual and Methodological Synergies

弥合分布偏移与AI安全：概念与方法论的协同

Chenruo Liu, Kenan Tang, Yao Qin, Qi Lei

机构 * Center for Data Science, New York University New York New York USA ； Computer Science Department, University of California, Santa Barbara Santa Barbara California USA ； Department of Electrical ； Computer Engineering, University of California, Santa Barbara Santa Barbara California USA ； Courant Institute for Mathematical Sciences \& Center for Data Science, New York University New York New York USA ； Center for Data Science, New York University ； Computer Science Department, University of California, Santa Barbara ； Computer Engineering, University of California, Santa Barbara ； Courant Institute for Mathematical Sciences \& Center for Data Science, New York University 纠错

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；分类 cs.AI、cs.LG

AI总结 本文通过分析分布偏移与AI安全之间的概念和方法论协同，建立了特定偏移类型与细粒度安全问题之间的两种联系，促进了两领域研究的深度融合。

Comments 35 pages

LSSF: Safety Alignment for Large Language Models through Low-Rank Safety Subspace Fusion

LSSF: 通过低秩安全子空间融合实现大语言模型的安全对齐

Guanghao Zhou, Panjia Qiu, Cen Chen, Hongyu Li, Mingyuan Chu, Xin Zhang, Jun Zhou

机构 * East China Normal University（华东师范大学） ； Ant Group（蚂蚁集团） 纠错

专题命中 其他安全 ：alignment(title,abstract)；safety(title,abstract)；分类 cs.AI、cs.CY

AI总结 LSSF通过低秩安全子空间融合技术，有效恢复微调大语言模型的安全对齐，同时对性能影响较小。

Comments Accepted in ACL 2025 Main Conference

Safety-Utility Conflicts Are Not Global: Surgical Alignment via Head-Level Diagnosis

安全与效用冲突并非全球性：通过头部层面诊断的手术对齐

Wang Cai, Yilin Wen, Jinchang Hou, Du Su, Guoqiu Wang, Zhonghou Lv, Chenfu Bao, Yunfang Wu

机构 * Baidu Inc.（百度公司） ； School of Software and Microelectronics, Peking University（北京大学软件与微电子学院） ； School of Computer Science, Peking University（北京大学计算机学院） ； State Key Laboratory of AI Safety, Institute of Computing Technology, CAS（中国科学院计算技术研究所人工智能安全重点实验室） 纠错

专题命中 其他安全 ：alignment(title,abstract)；safety(title,abstract)；分类 cs.AI、cs.LG

AI总结 本文提出CAST框架，通过头部层面诊断与稀疏微调相结合，解决LLMs中安全与效用冲突问题，减少训练损失并提升安全性。

Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety

思维链可监控性：AI安全的新且脆弱的机会

Tomek Korbak, Mikita Balesni, Elizabeth Barnes, Yoshua Bengio, Joe Benton, Joseph Bloom, Mark Chen, Alan Cooney, Allan Dafoe, Anca Dragan, Scott Emmons, Owain Evans, David Farhi, Ryan Greenblatt, Dan Hendrycks, Marius Hobbhahn, Evan Hubinger, Geoffrey Irving, Erik Jenner, Daniel Kokotajlo, Victoria Krakovna, Shane Legg, David Lindner, David Luan, Aleksander Mądry, Julian Michael, Neel Nanda, Dave Orr, Jakub Pachocki, Ethan Perez, Mary Phuong, Fabien Roger, Joshua Saxe, Buck Shlegeris, Martín Soto, Eric Steinberger, Jasmine Wang, Wojciech Zaremba, Bowen Baker, Rohin Shah, Vlad Mikulik

机构 * UK AI Security Institute（英国人工智能安全研究所） ； Apollo Research（阿波罗研究） ； METR ； University of Montreal（蒙特利尔大学） ； Mila ； Anthropic ； OpenAI（开放人工智能研究所） ； Google DeepMind（谷歌DeepMind） ； Truthful AI ； UC Berkeley（伯克利大学） ； Center for AI Safety（人工智能安全中心） ； AI Futures Project（人工智能未来项目） ； Amazon（亚马逊） ； Scale AI ； Magic ； Meta ； Redwood Research（红木研究） 纠错

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；分类 cs.AI、cs.LG

AI总结 本文探讨了通过监控AI思维链来提升安全性的潜力，指出其脆弱性并呼吁进一步研究和投资。

Annotating the Chain-of-Thought: A Behavior-Labeled Dataset for AI Safety

Antonio-Gabriel Chacón Menke, Phan Xuan Tan, Eiji Kamioka

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；分类 cs.AI、cs.CY

SLM as Guardian: Pioneering AI Safety with Small Language Models

Ohjoon Kwon, Donghyeon Jeon, Nayoung Choi, Gyu-Hwung Cho, Changbong Kim, Hyunwoo Lee, Inho Kang, Sun Kim, Taiwoo Park

专题命中 其他安全 ：safety(title,abstract)；AI safety(title)；alignment(abstract)；分类 cs.CL、cs.AI

Upstream and Downstream AI Safety: Both on the Same River?

John McDermid, Yan Jia, Ibrahim Habli

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；分类 cs.AI、cs.CY

Landscape of AI safety concerns -- A methodology to support safety assurance for AI-based autonomous systems

Ronald Schnitzer, Lennart Kilian, Simon Roessner, Konstantinos Theodorou, Sonja Zillner

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；分类 cs.AI、cs.LG

The Elephant in the Room -- Why AI Safety Demands Diverse Teams

David Rostcheck, Lara Scheibling

专题命中 其他安全 ：safety(title,abstract)；AI safety(title)；alignment(abstract)；分类 cs.AI、cs.CY

Comments 9 pages

Concrete Problems in AI Safety, Revisited

Inioluwa Deborah Raji, Roel Dobbe

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；分类 cs.AI、cs.CY

Comments Published at ICLR workshop on ML in the Real World, 2020

GR-SAP: Generative Replay for Safety Alignment Preservation during Fine-Tuning

GR-SAP: 生成性重放用于在微调过程中保持安全对齐

Zhouxiang Fang, Jiawei Zhou, Hanjie Chen

专题命中 其他安全 ：alignment(title,abstract)；safety(title,abstract)；分类 cs.CL

AI总结 GR-SAP通过生成领域特定对齐数据来保持微调过程中的安全对齐，有效缓解了微调导致的安全退化问题。

AI Safety Evaluations Need To Consider Cascading Effects

AI安全评估需要考虑连锁效应

Anna Neumann, Jatinder Singh

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；分类 cs.CY

AI总结 本文提出通过分析AI供应链中各组件的连锁效应，改进AI安全评估方法，以提升系统透明度和安全性。

Comments As accepted in the IASEAI 2026 Proceedings, Track 11: Epistemological challenges of AI

International AI Safety Report 2026

2026国际人工智能安全报告

Yoshua Bengio, Stephen Clare, Carina Prunkl, Maksym Andriushchenko, Ben Bucknall, Malcolm Murray, Rishi Bommasani, Stephen Casper, Tom Davidson, Raymond Douglas, David Duvenaud, Philip Fox, Usman Gohar, Rose Hadshar, Anson Ho, Tiancheng Hu, Cameron Jones, Sayash Kapoor, Atoosa Kasirzadeh, Sam Manning, Nestor Maslej, Vasilios Mavroudis, Conor McGlynn, Richard Moulange, Jessica Newman, Kwan Yee Ng, Patricia Paskov, Shalaleh Rismani, Girish Sastry, Elizabeth Seger, Scott Singer, Charlotte Stix, Lucia Velasco, Nicole Wheeler, Daron Acemoglu, Vincent Conitzer, Thomas G. Dietterich, Fredrik Heintz, Geoffrey Hinton, Nick Jennings, Susan Leavy, Teresa Ludermir, Vidushi Marda, Helen Margetts, John McDermid, Jane Munga, Arvind Narayanan, Alondra Nelson, Clara Neppel, Sarvapali D. Ramchurn, Stuart Russell, Marietje Schaake, Bernhard Schölkopf, Alvaro Soto, Lee Tiedrich, Gaël Varoquaux, Andrew Yao, Ya-Qin Zhang, Leandro Angelo Aguirre, Olubunmi Ajala, Fahad Albalawi, Noora AlMalek, Christian Busch, Jonathan Collas, André Carlos Ponce de Leon Ferreira de Carvalho, Amandeep Gill, Ahmet Halit Hatip, Juha Heikkilä, Chris Johnson, Gill Jolly, Ziv Katzir, Mary N. Kerema, Hiroaki Kitano, Antonio Krüger, Kyoung Mu Lee, José Ramón López Portillo, Aoife McLysaght, Oleksii Molchanovskyi, Andrea Monti, Mona Nemer, Nuria Oliver, Raquel Pezoa, Audrey Plonk, Balaraman Ravindran, Hammam Riza, Crystal Rugege, Haroon Sheikh, Denise Wong, Yi Zeng, Liming Zhu, Daniel Privitera, Sören Mindermann

专题命中 其他安全 ：safety(title,abstract)；AI safety(title,abstract)；分类 cs.CY

AI总结 2026国际人工智能安全报告由全球29国及国际组织共同参与，汇集超过100位专家意见，全面评估通用人工智能系统的安全性和潜在风险。

Personalized Safety Alignment for Text-to-Image Diffusion Models

面向文本到图像扩散模型的个性化安全对齐

Yu Lei, Jinbin Bai, Qingyu Shi, Aosong Feng, Hongcheng Gao, Xiao Zhang, Rex Ying

机构 * National University of Singapore（新加坡国立大学） ； Peking University（北京大学） ； Yale University（耶鲁大学） ； Collov Labs（Collov实验室） 纠错

专题命中 其他安全 ：alignment(title,abstract)；safety(title,abstract)；分类 cs.AI

AI总结 本文提出个性化安全对齐框架PSA，通过用户条件适应提升文本到图像扩散模型的安全性与生成质量，实现安全与质量的动态平衡。