大模型对齐与安全

LLMail-Inject: A Dataset from a Realistic Adaptive Prompt Injection Challenge

Sahar Abdelnabi, Aideen Fay, Ahmed Salem, Egor Zverev, Kai-Chieh Liao, Chi-Huang Liu, Chun-Chih Kuo, Jannis Weigend, Danyael Manlangit, Alex Apostolov, Haris Umair, João Donato, Masayuki Kawakita, Athar Mahboob, Tran Huu Bach, Tsun-Han Chiang, Myeongjin Cho, Hajin Choi, Byeonghyeon Kim, Hyeonjin Lee, Benjamin Pannell, Conor McCauley, Mark Russinovich, Andrew Paverd, Giovanni Cherubin

机构 * Microsoft（微软） ； ISTA ； Trend Micro ； RainaResearch ； University of Coimbra（科英布拉大学） ； Vietnamese German University（越南-德国大学） ； SK Shieldus ； HiddenLayer 纠错

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments Dataset at: https://huggingface.co/datasets/microsoft/llmail-inject-challenge

MELON: Provable Defense Against Indirect Prompt Injection Attacks in AI Agents

Kaijie Zhu, Xianjun Yang, Jindong Wang, Wenbo Guo, William Yang Wang

机构 * University of California, Santa Barbara（加州大学圣巴巴拉分校） 纠错

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments ICML 2025

Alignment Drift in CEFR-prompted LLMs for Interactive Spanish Tutoring

Mina Almasi, Ross Deans Kristensen-McLachlan

机构 * Department of Linguistics, Cognitive Science, and Semiotics（语言学、认知科学与符号学系） 纠错

专题命中 提示注入 ：alignment(title,abstract)；分类 cs.CL

Comments Accepted at BEA2025 (Conference workshop at ACL 2025)

To Protect the LLM Agent Against the Prompt Injection Attack with Polymorphic Prompt

Zhilong Wang, Neha Nagaraja, Lan Zhang, Hayretdin Bahsi, Pawan Patil, Peng Liu

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments To appear in the Industry Track of the 55th Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN 2025)

Sentinel: SOTA model to protect against prompt injections

Dror Ivry, Oran Nahum

机构 * Qualifire 纠错

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments 6 pages, 2 tables

A Critical Evaluation of Defenses against Prompt Injection Attacks

Yuqi Jia, Zedian Shao, Yupei Liu, Jinyuan Jia, Dawn Song, Neil Zhenqiang Gong

机构 * Duke University（杜克大学） ； The Pennsylvania State University（宾夕法尼亚州立大学） ； UC Berkeley（伯克利大学） 纠错

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

WASP: Benchmarking Web Agent Security Against Prompt Injection Attacks

Ivan Evtimov, Arman Zharmagambetov, Aaron Grattafiori, Chuan Guo, Kamalika Chaudhuri

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments Code and data: https://github.com/facebookresearch/wasp

Fun-tuning: Characterizing the Vulnerability of Proprietary LLMs to Optimization-based Prompt Injection Attacks via the Fine-Tuning Interface

Andrey Labunets, Nishit V. Pandya, Ashish Hooda, Xiaohan Fu, Earlence Fernandes

机构 * UC San Diego（加州大学圣地亚哥分校） ； University of Wisconsin Madison（威斯康星大学麦迪逊分校） 纠错

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

Journal ref Proceedings of the 2025 IEEE Symposium on Security and Privacy, IEEE Computer Society, 2025, pp. 374-392

Adversarial Attacks on LLM-as-a-Judge Systems: Insights from Prompt Injections

Narek Maloyan, Dmitry Namiot

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

PROMPTFUZZ: Harnessing Fuzzing Techniques for Robust Testing of Prompt Injection in LLMs

Jiahao Yu, Yangguang Shao, Hanwen Miao, Junzheng Shi

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Adaptive Attacks Break Defenses Against Indirect Prompt Injection Attacks on LLM Agents

Qiusi Zhan, Richard Fang, Henil Shalin Panchal, Daniel Kang

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.LG

Comments 17 pages, 5 figures, 6 tables (NAACL 2025 Findings)

RTBAS: Defending LLM Agents Against Prompt Injection and Privacy Leakage

Peter Yong Zhong, Siyuan Chen, Ruiqi Wang, McKenna McCall, Ben L. Titzer, Heather Miller, Phillip B. Gibbons

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

FATH: Authentication-based Test-time Defense against Indirect Prompt Injection Attacks

Jiongxiao Wang, Fangzhou Wu, Wendi Li, Jinsheng Pan, Edward Suh, Z. Morley Mao, Muhao Chen, Chaowei Xiao

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents

Edoardo Debenedetti, Jie Zhang, Mislav Balunović, Luca Beurer-Kellner, Marc Fischer, Florian Tramèr

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.LG

Comments Updated version after fixing a bug in the Llama implementation and updating the travel suite

A test suite of prompt injection attacks for LLM-based machine translation

Antonio Valerio Miceli-Barone, Zhifan Sun

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

Hacking Back the AI-Hacker: Prompt Injection as a Defense Against LLM-driven Cyberattacks

Dario Pasquini, Evgenios M. Kornaropoulos, Giuseppe Ateniese

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments v0.2 (evaluated on more agents)

Embedding-based classifiers can detect prompt injection attacks

Md. Ahsan Ayub, Subhabrata Majumdar

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.LG

Backdoored Retrievers for Prompt Injection Attacks on Retrieval Augmented Generation of Large Language Models

Cody Clop, Yannick Teglia

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.LG

Comments 12 pages, 5 figures

F2A: An Innovative Approach for Prompt Injection by Utilizing Feign Security Detection Agents

Yupeng Ren

专题命中 提示注入 ：prompt injection(title)；safety(abstract)；分类 cs.AI

Comments 1. Fixed typo in abstract 2. Provisionally completed the article update to facilitate future version revisions

Applying Pre-trained Multilingual BERT in Embeddings for Improved Malicious Prompt Injection Attacks Detection

Md Abdur Rahman, Hossain Shahriar, Fan Wu, Alfredo Cuzzocrea

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

InjecAgent: Benchmarking Indirect Prompt Injections in Tool-Integrated Large Language Model Agents

Qiusi Zhan, Zhixiang Liang, Zifan Ying, Daniel Kang

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

Comments 36 pages, 6 figures, 13 tables (ACL 2024 Findings)

Scaling Behavior of Machine Translation with Large Language Models under Prompt Injection Attacks

Zhifan Sun, Antonio Valerio Miceli-Barone

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.CL

Comments 15 pages, 18 figures, First Workshop on the Scaling Behavior of Large Language Models (SCALE-LLM 2024)

Automatic and Universal Prompt Injection Attacks against Large Language Models

Xiaogeng Liu, Zhiyuan Yu, Yizhe Zhang, Ning Zhang, Chaowei Xiao

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Comments Pre-print, code is available at https://github.com/SheltonLiu-N/Universal-Prompt-Injection

Signed-Prompt: A New Approach to Prevent Prompt Injection Attacks Against LLM-Integrated Applications

Xuchen Suo

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.AI

Tensor Trust: Interpretable Prompt Injection Attacks from an Online Game

Sam Toyer, Olivia Watkins, Ethan Adrian Mendes, Justin Svegliato, Luke Bailey, Tiffany Wang, Isaac Ong, Karim Elmaaroufi, Pieter Abbeel, Trevor Darrell, Alan Ritter, Stuart Russell

专题命中 提示注入 ：prompt injection(title,abstract)；分类 cs.LG

StruQ: Defending Against Prompt Injection with Structured Queries

Sizhe Chen, Julien Piet, Chawin Sitawarin, David Wagner

专题命中 提示注入 ：prompt injection(title,abstract)

Comments To appear at USENIX Security Symposium 2025. Key words: prompt injection defense, LLM security, LLM-integrated applications

Countering Autonomous Cyber Threats

Kade M. Heckel, Adrian Weller

专题命中 提示注入 ：safety(abstract)；prompt injection(abstract)；AI safety(abstract)；分类 cs.AI、cs.CY

Comments 76 pages, MPhil Thesis

Attention is All You Need to Defend Against Indirect Prompt Injection Attacks in LLMs

注意力是所有你需要的：用于防御大语言模型中的间接提示注入攻击

Yinan Zhong, Qianhao Miao, Yanjiao Chen, Jiangyi Deng, Yushi Cheng, Wenyuan Xu

专题命中 提示注入 ：prompt injection(title,abstract)

AI总结 本文提出Rennervate框架，通过注意力机制在token级别检测并清除IPI攻击，有效提升LLM的安全性。

Comments Accepted by Network and Distributed System Security (NDSS) Symposium 2026

PromptShield Home: Ambient Multimodal Prompt Injection Defense for Smart-Home Agents

PromptShield Home：面向智能家居智能体的环境多模态提示注入防御方案

He Zhang, Feilong Li, Dingning Long, Yilin Cui, Peijun Zhang, Yuewen Zhang, Qianyao Xu, Xinyi Fu

专题命中 提示注入 ：prompt injection(title)；safety(abstract)

AI总结 该研究针对智能家居智能体的提示注入安全问题，构建了现实场景基准测试PromptShield-Home，对比了三类防御层的性能，发现多智能体调解的上限性能最优，提出应采用学习路由与传感器融合方案保障安全。

Comments This work has been accepted as a poster to UbiComp 2026

Piggybacking on Perception: Stealthy Concurrent Audio Prompt Injections against Multimodal LLM Agents

利用感知能力：针对多模态大语言模型智能体的隐蔽并发音频提示注入攻击

Mingxiao Liu, Yitong Li, Haoren Zhao, Yaoxiang Bian, Jianan Ma, Jian Zhang, Jialuo Chen, Xinhao Deng, Zhen Wang

专题命中 提示注入 ：prompt injection(title,abstract)

AI总结 该研究针对多模态LLM智能体提出隐蔽并发音频提示注入攻击，构建首个相关基准并评估多款智能体，还提出CADV防御机制，经实验验证攻击有效且防御可靠。

Comments 19 pages, 8 figures, The code is publicly available at https://github.com/Limax666/AudioAgentSecurity