大模型对齐与安全

TEMPLATEFUZZ: Fine-Grained Chat Template Fuzzing for Jailbreaking and Red Teaming LLMs

TEMPLATEFUZZ：细粒度聊天模板模糊测试用于突破和红队测试大语言模型

Qingchao Shen, Zibo Xiao, Lili Huang, Enwei Hu, Yongqiang Tian, Junjie Chen

机构 * School of Computer Software（计算机软件学院） ； Tianjin University（天津大学） ； Monash University（墨尔本大学） 纠错

专题命中 红队测试 ：red teaming(title)；safety(abstract)；jailbreak(abstract)；prompt injection(abstract)

AI总结 本文提出TEMPLATEFUZZ框架，通过细粒度模糊测试揭示大语言模型聊天模板的漏洞，实现高攻击成功率的同时保持模型准确性，优于现有方法。

Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planning

视觉排斥攻击：通过代理规划实现自动多模态红队行动

Yunbei Zhang, Yingqiang Ge, Weijie Xu, Yuhui Xu, Jihun Hamm, Chandan K. Reddy

专题命中 红队测试 ：red teaming(title,abstract)；alignment(abstract)；safety(abstract)；分类 cs.LG

AI总结 本文提出视觉排斥攻击，通过多模态多轮代理规划框架MM-Plan，实现基于视觉内容的攻击，展示了对前沿模型的高攻击成功率，揭示了当前安全对齐的不足。

AutoRed: A Free-form Adversarial Prompt Generation Framework for Automated Red Teaming

Muxi Diao, Yutao Mou, Keqing He, Hanbo Song, Lulu Zhao, Shikun Zhang, Wei Ye, Kongming Liang, Zhanyu Ma

机构 * Beijing University of Posts and Telecommunications（北京邮电大学） ； Peking University（北京大学） ； Beijing Academy of Artificial Intelligence（北京人工智能研究院） 纠错

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；trustworthy(abstract)；分类 cs.CL

Jinx: Unlimited LLMs for Probing Alignment Failures

Jiahao Zhao, Liwei Dong

专题命中 红队测试 ：alignment(title,abstract)；safety(abstract)；red teaming(abstract)；分类 cs.CL

Comments https://huggingface.co/Jinx-org

Multi-lingual Multi-turn Automated Red Teaming for LLMs

Abhishek Singhania, Christophe Dupuy, Shivam Mangale, Amani Namboori

专题命中 红队测试 ：red teaming(title,abstract)；alignment(abstract)；safety(abstract)；分类 cs.CL

Comments Accepted at TrustNLP@NAACL 2025

Effective Automation to Support the Human Infrastructure in AI Red Teaming

Alice Qian Zhang, Jina Suh, Mary L. Gray, Hong Shen

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；AI safety(abstract)；分类 cs.CY

Comments This piece has been accepted to the ACM Interactions Publication Tech Labor Forum For August 2025

Red Teaming Contemporary AI Models: Insights from Spanish and Basque Perspectives

Miguel Romero-Arjona, Pablo Valle, Juan C. Alonso, Ana B. Sánchez, Miriam Ugarte, Antonia Cazalilla, Vicente Cambrón, José A. Parejo, Aitor Arrieta, Sergio Segura

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；trustworthy(abstract)；分类 cs.CL

Gradient-Based Language Model Red Teaming

Nevan Wichers, Carson Denison, Ahmad Beirami

专题命中 红队测试 ：red teaming(title,abstract)；alignment(abstract)；safety(abstract)；分类 cs.CL

Comments EACL 2024 main conference

GPT-Red: Automated Red Teaming via Self-Play at Scale

GPT-Red：基于大规模自博弈的自动化红队测试

Eric Wallace, Christopher A. Choquette-Choo, Nikhil Kandpal, Sam Toyer, Dylan Hunn, Stephanie Lin, Yuxin Wen, Xiangyu Qi, Christopher Wolff, Zizhao Wang, Milad Nasr, Sicheng Zhu, Chuan Guo, Juan Felipe Cerón Uribe, Kaiwen Wang, Aiden Low, Kai Xiao, Kai Chen

专题命中 红队测试 ：red teaming(title)；safety(abstract)；prompt injection(abstract)；分类 cs.CL、cs.AI、cs.LG

AI总结 本研究推出GPT-Red，一种基于大规模自博弈的自动化红队测试智能体，可发现新型提示注入攻击、攻破过往模型且泛化能力强，用于提升LLM鲁棒性并形成自我改进飞轮。

Comments 28 pages.13 main pages and 13 main figures

Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming

评估大型语言模型在心理健康支持中的风险：一种用于自动化临床AI红队测试的框架

Ian Steenstra, Paola Pedrelli, Weiyan Shi, Stacy Marsella, Timothy W. Bickmore

机构 * Northeastern University（东北大学） ； Harvard Medical School（哈佛医学院） 纠错

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.CY

AI总结 本文提出了一种评估AI心理治疗师在心理健康支持中安全风险的框架，通过模拟测试发现AI在治疗中的潜在风险，并验证了交互式可视化工具的有效性。

Comments This paper is a condensed version of the first author's Ph.D. dissertation submitted to Northeastern University

Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluation

对LLM进行红队行动作为社会技术实践：从探索和数据创建到评估

Adriana Alvarado Garcia, Ruyuan Wan, Ozioma C. Oguine, Karla Badillo-Urquiola

机构 * Responsible Tech Research IBM Research Yorktown Heights New York USA（负责技术研究 IBM 研究 Yorktown Heights 新 York 美国） ； IBM Research（IBM 研究） ； The Pennsylvania State University（宾夕法尼亚州立大学） ； University of Notre Dame（诺特达美大学） 纠错

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.CY

AI总结 本文通过访谈从业者，探讨了红队行动数据集定义、创建和评估的社会技术实践，揭示了从业者在风险概念化中的不足，并提出HCI研究者扩展红队行动概念化和数据实践的三个机会。

Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming

Mrinank Sharma, Meg Tong, Jesse Mu, Jerry Wei, Jorrit Kruthoff, Scott Goodfriend, Euan Ong, Alwin Peng, Raj Agarwal, Cem Anil, Amanda Askell, Nathan Bailey, Joe Benton, Emma Bluemke, Samuel R. Bowman, Eric Christiansen, Hoagy Cunningham, Andy Dau, Anjali Gopal, Rob Gilson, Logan Graham, Logan Howard, Nimit Kalra, Taesung Lee, Kevin Lin, Peter Lofgren, Francesco Mosconi, Clare O'Hara, Catherine Olsson, Linda Petrini, Samir Rajani, Nikhil Saxena, Alex Silverstein, Tanya Singh, Theodore Sumers, Leonard Tang, Kevin K. Troy, Constantin Weisser, Ruiqi Zhong, Giulio Zhou, Jan Leike, Jared Kaplan, Ethan Perez

专题命中 红队测试 ：red teaming(title,abstract)；jailbreak(abstract)；分类 cs.CL、cs.AI、cs.LG

Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning

Alex Beutel, Kai Xiao, Johannes Heidecke, Lilian Weng

专题命中 红队测试 ：red teaming(title,abstract)；prompt injection(abstract)；分类 cs.CL、cs.AI、cs.LG

DiveR-CT: Diversity-enhanced Red Teaming Large Language Model Assistants with Relaxing Constraints

Andrew Zhao, Quentin Xu, Matthieu Lin, Shenzhi Wang, Yong-jin Liu, Zilong Zheng, Gao Huang

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.LG

Comments Accepted by the 39th Annual AAAI Conference on Artificial Intelligence (AAAI-25)

STAR: SocioTechnical Approach to Red Teaming Language Models

Laura Weidinger, John Mellor, Bernat Guillen Pegueroles, Nahema Marchal, Ravin Kumar, Kristian Lum, Canfer Akbulut, Mark Diaz, Stevie Bergman, Mikel Rodriguez, Verena Rieser, William Isaac

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI、cs.CY

Comments 8 pages, 5 figures, 5 pages appendix. * denotes equal contribution

Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned

Deep Ganguli, Liane Lovitt, Jackson Kernion, Amanda Askell, Yuntao Bai, Saurav Kadavath, Ben Mann, Ethan Perez, Nicholas Schiefer, Kamal Ndousse, Andy Jones, Sam Bowman, Anna Chen, Tom Conerly, Nova DasSarma, Dawn Drain, Nelson Elhage, Sheer El-Showk, Stanislav Fort, Zac Hatfield-Dodds, Tom Henighan, Danny Hernandez, Tristan Hume, Josh Jacobson, Scott Johnston, Shauna Kravec, Catherine Olsson, Sam Ringer, Eli Tran-Johnson, Dario Amodei, Tom Brown, Nicholas Joseph, Sam McCandlish, Chris Olah, Jared Kaplan, Jack Clark

专题命中 红队测试 ：red teaming(title,abstract)；RLHF(abstract)；分类 cs.CL、cs.AI、cs.CY

Security practices in AI development

Petr Spelda, Vit Stritecky

专题命中 红队测试 ：alignment(abstract)；safety(abstract)；red teaming(abstract)；trustworthy(abstract)

Comments 11 pages

A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation

大型语言模型的红队框架：以忠实性评估为例

Abrar Alotaibi, Raed Mughus, Moataz Ahmed

机构 * King Fahd University of Petroleum & Minerals（法赫德国王石油矿产大学） ； Imam Abdulrahman Bin Faisal University（伊玛目阿卜杜勒拉赫曼·本·费萨尔大学） ； SDAIA-KFUPM Joint Research Center for Artificial Intelligence（沙特数据与人工智能局-法赫德国王石油矿产大学人工智能联合研究中心） 纠错

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；分类 cs.CL、cs.AI

AI总结 提出一种多角色红队框架，通过攻击者和陪审模型暴露LLM的不忠实问题，在问答任务中攻击成功率提升7.9%，并揭示结构约束和架构设计对安全性的影响。

Comments Preprint submitted to SQJ

Owner-Harm: A Missing Threat Model for AI Agent Safety

所有权危害：AI代理安全的缺失威胁模型

Dongcheng Zhang, Yiqing Jiang

机构 * BlueFocus Communication Group（蓝ocus通信集团） ； Tongji University（同济大学） 纠错

专题命中 红队测试 ：safety(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

AI总结 本文提出Owner-Harm威胁模型，旨在解决AI代理对部署者造成伤害的问题，通过实验验证了现有防御体系的不足，并引入SSDG框架提升检测效果。

Comments 15 pages. Companion manuscript on per-decision proof-obligation synthesis (LSVJ-S) in preparation

A Safety and Security Framework for Real-World Agentic Systems

面向现实世界代理系统的安全与安全框架

Shaona Ghosh, Barnaby Simkin, Kyriacos Shiarlis, Soumili Nandi, Dan Zhao, Matthew Fiedler, Julia Bazinska, Nikki Pope, Roopa Prabhu, Daniel Rohrer, Michael Demoret, Bartley Richardson

机构 * NVIDIA ； Lakera AI 纠错

专题命中 红队测试 ：safety(title,abstract)；red teaming(abstract)；分类 cs.AI、cs.LG

AI总结 本文提出了一种动态框架，通过用户安全视角识别新型代理风险，并通过红队行动和案例研究验证其在企业级代理系统中的安全性和安全性。

Red Teaming for Generative AI, Report on a Copyright-Focused Exercise Completed in an Academic Medical Center

James Wen, Sahil Nalawade, Zhiwei Liang, Catherine Bielick, Marisa Ferrara Boston, Alexander Chowdhury, Adele Collin, Luigi De Angelis, Jacob Ellen, Heather Frase, Rodrigo R. Gameiro, Juan Manuel Gutierrez, Pooja Kadam, Murat Keceli, Srikanth Krishnamurthy, Anne Kwok, Yanan Lance Lu, Heather Mattie, Liam G. McCoy, Katherine Miller, Allison C. Morgan, Marlene Louisa Moerig, Trang Nguyen, Alexander Owen-Post, Alex D. Ruiz, Sreekar Reddy Puchala, Soujanya Samineni, Takeshi Tohyama, Varun Ullanat, Carmine Valenza, Camilo Velez, Pengcheng Wang, Anna Wuest, Yuxiang Zhou, Yingde Zhu, Jason M. Johnson, Naomi Lenane, Jennifer Willcox, Francis J. Vitiello, Leo Anthony G. Celi, Renato Umeton

专题命中 红队测试 ：red teaming(title,abstract)；jailbreak(abstract)；分类 cs.AI、cs.CY

We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systems

Junfeng Fang, Zijun Yao, Ruipeng Wang, Haokai Ma, Xiang Wang, Tat-Seng Chua

机构 * National University of Singapore（新加坡国立大学） ； University of Science and Technology of China（中国科学技术大学） 纠错

专题命中 红队测试 ：safety(title,abstract)；red teaming(abstract)；分类 cs.AI、cs.LG

OpenAI's Approach to External Red Teaming for AI Models and Systems

Lama Ahmad, Sandhini Agarwal, Michael Lampe, Pamela Mishkin

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；分类 cs.AI、cs.CY

Embodied Red Teaming for Auditing Robotic Foundation Models

Sathwik Karnik, Zhang-Wei Hong, Nishant Abhangi, Yen-Chen Lin, Tsun-Hsuan Wang, Christophe Dupuy, Rahul Gupta, Pulkit Agrawal

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；分类 cs.AI、cs.LG

Automated Red Teaming with GOAT: the Generative Offensive Agent Tester

Maya Pavlova, Erik Brinkman, Krithika Iyer, Vitor Albiero, Joanna Bitton, Hailey Nguyen, Joe Li, Cristian Canton Ferrer, Ivan Evtimov, Aaron Grattafiori

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；分类 cs.AI、cs.LG

Holistic Automated Red Teaming for Large Language Models through Top-Down Test Case Generation and Multi-turn Interaction

Jinchuan Zhang, Yan Zhou, Yaxin Liu, Ziming Li, Songlin Hu

专题命中 红队测试 ：red teaming(title,abstract)；alignment(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2024 camera ready version

Phi-3 Safety Post-Training: Aligning Language Models with a "Break-Fix" Cycle

Emman Haider, Daniel Perez-Becker, Thomas Portet, Piyush Madan, Amit Garg, Atabak Ashfaq, David Majercak, Wen Wen, Dongwoo Kim, Ziyi Yang, Jianwen Zhang, Hiteshi Sharma, Blake Bullwinkel, Martin Pouliot, Amanda Minnich, Shiven Chawla, Solianna Herrera, Shahed Warreth, Maggie Engler, Gary Lopez, Nina Chikanov, Raja Sekhar Rao Dheekonda, Bolor-Erdene Jagdagdorj, Roman Lutz, Richard Lundeen, Tori Westerhoff, Pete Bryan, Christian Seifert, Ram Shankar Siva Kumar, Andrew Berkley, Alex Kessler

专题命中 红队测试 ：safety(title,abstract)；red teaming(abstract)；分类 cs.CL、cs.AI

Attack Prompt Generation for Red Teaming and Defending Large Language Models

Boyi Deng, Wenjie Wang, Fuli Feng, Yang Deng, Qifan Wang, Xiangnan He

专题命中 红队测试 ：red teaming(title,abstract)；safety(abstract)；分类 cs.CL、cs.LG

Comments Accepted to EMNLP 2023 (Findings)

Kov: Transferable and Naturalistic Black-Box LLM Attacks using Markov Decision Processes and Tree Search

Robert J. Moss

专题命中 红队测试 ：alignment(abstract)；safety(abstract)；jailbreak(abstract)；red teaming(abstract)

From Inaudible Inputs to Model Failures: Low-Frequency Safety Risks in LALMs

从不可听输入到模型失效：LALMs中的低频安全风险

Yuanhe Zhang, Weiliu Wang, Jie Ren, Liang Lin, Zhenhong Zhou, Haoran Gao, Kun Wang, Chen Li, Li Sun, Sen Su

机构 * Beijing University of Posts and Telecommunications（北京邮电大学） ； Institute of Information Engineering, Chinese Academy of Sciences（中国科学院信息工程研究所） ； Nanyang Technological University（南洋理工大学） ； JIUTIAN Research（九天研究院） ； Tencent ARC Lab（腾讯ARC实验室） ； Chongqing University of Posts and Telecommunications（重庆邮电大学） 纠错

专题命中 红队测试 ：safety(title,abstract)；red teaming(abstract)；分类 cs.AI

AI总结 本文提出不可听红队测试方法ILL评估LALMs的低频安全风险，发现其可降低LALMs准确率达67个百分点，同时提出DRG防护方法可提升受攻击后的准确率，明确了LALMs此前被忽视的安全风险。