视觉大模型 / VLM

MBQ: Modality-Balanced Quantization for Large Vision-Language Models

Shiyao Li, Yingchun Hu, Xuefei Ning, Xihui Liu, Ke Hong, Xiaotao Jia, Xiuhong Li, Yaqi Yan, Pei Ran, Guohao Dai, Shengen Yan, Huazhong Yang, Yu Wang

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoders

Qing Li, Jiahui Geng, Derui Zhu, Fengyu Cai, Chenyang Lyu, Fakhri Karray

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Comments More comparative experiments are needed

Towards Statistical Factuality Guarantee for Large Vision-Language Models

Zhuohang Li, Chao Yan, Nicholas J. Jackson, Wendi Cui, Bo Li, Jiaxin Zhang, Bradley A. Malin

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.LG

Tracking the Copyright of Large Vision-Language Models through Parameter Learning Adversarial Images

Yubo Wang, Jianting Tang, Chaohu Liu, Linli Xu

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.AI、cs.LG

Comments Accepted to ICLR 2025

EigenShield: Causal Subspace Filtering via Random Matrix Theory for Adversarially Robust Vision-Language Models

Nastaran Darabi, Devashri Naik, Sina Tayebati, Dinithi Jayasuriya, Ranganath Krishnan, Amit Ranjan Trivedi

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Learning Conformal Abstention Policies for Adaptive Risk Management in Large Language and Vision-Language Models

Sina Tayebati, Divake Kumar, Nastaran Darabi, Dinithi Jayasuriya, Ranganath Krishnan, Amit Ranjan Trivedi

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.AI、cs.LG

ETA: Evaluating Then Aligning Safety of Vision Language Models at Inference Time

Yi Ding, Bolian Li, Ruqi Zhang

专题命中 幻觉与鲁棒性 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments 29pages

Advancing the Understanding and Evaluation of AR-Generated Scenes: When Vision-Language Models Shine and Stumble

Lin Duan, Yanming Xiu, Maria Gorlatova

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments 6 pages

UniGuard: Towards Universal Safety Guardrails for Jailbreak Attacks on Multimodal Large Language Models

Sejoon Oh, Yiqiao Jin, Megha Sharma, Donghyun Kim, Eric Ma, Gaurav Verma, Srijan Kumar

专题命中 幻觉与鲁棒性 ：multimodal large language model(title,abstract)；LLaVA(abstract)；分类 cs.AI、cs.LG

Comments 14 pages

Black-Box Adversarial Attack on Vision Language Models for Autonomous Driving

Lu Wang, Tianyuan Zhang, Yang Qu, Siyuan Liang, Yuwei Chen, Aishan Liu, Xianglong Liu, Dacheng Tao

专题命中 幻觉与鲁棒性 ：vision language model(title)；vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Read Like a Radiologist: Efficient Vision-Language Model for 3D Medical Imaging Interpretation

Changsun Lee, Sangjoon Park, Cheong-Il Shin, Woo Hee Choi, Hyun Jeong Park, Jeong Eun Lee, Jong Chul Ye

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

More Distinctively Black and Feminine Faces Lead to Increased Stereotyping in Vision-Language Models

Messi H. J. Lee, Jacob M. Montgomery, Calvin K. Lai

专题命中 幻觉与鲁棒性 ：vision-language model(title)；vision language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments This submission is being withdrawn to address concerns related to the terms of use of a database utilized in the research. We aim to ensure full compliance with all data usage agreements before proceeding with publication

ViSTa Dataset: Do vision-language models understand sequential tasks?

Evžen Wybitul, Evan Ryan Gunter, Mikhail Seleznyov, David Lindner

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Mitigating Hallucination in Multimodal Large Language Model via Hallucination-targeted Direct Preference Optimization

Yuhan Fu, Ruobing Xie, Xingwu Sun, Zhanhui Kang, Xirong Li

专题命中 幻觉与鲁棒性 ：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

VHELM: A Holistic Evaluation of Vision Language Models

Tony Lee, Haoqin Tu, Chi Heem Wong, Wenhao Zheng, Yiyang Zhou, Yifan Mai, Josselin Somerville Roberts, Michihiro Yasunaga, Huaxiu Yao, Cihang Xie, Percy Liang

专题命中 幻觉与鲁棒性 ：vision language model(title,abstract)；vision-language model(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2024. First three authors contributed equally

Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks

Md Zarif Hossain, Ahmed Imteaj

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation

Xiaoye Qu, Qiyuan Chen, Wei Wei, Jishuo Sun, Jianfeng Dong

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Arondight: Red Teaming Large Vision Language Models with Auto-generated Multi-modal Jailbreak Prompts

Yi Liu, Chengjun Cai, Xiaoli Zhang, Xingliang Yuan, Cong Wang

专题命中 幻觉与鲁棒性 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.AI、cs.LG

Comments To be published in ACM MM 2024

GenderBias-\emph{VL}: Benchmarking Gender Bias in Vision Language Models via Counterfactual Probing

Yisong Xiao, Aishan Liu, QianJia Cheng, Zhenfei Yin, Siyuan Liang, Jiapeng Li, Jing Shao, Xianglong Liu, Dacheng Tao

专题命中 幻觉与鲁棒性 ：vision language model(title)；vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 4 figures

Uncovering Bias in Large Vision-Language Models with Counterfactuals

Phillip Howard, Anahita Bhiwandiwalla, Kathleen C. Fraser, Svetlana Kiritchenko

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Accepted to the CVPR 2024 Responsible Generative AI (ReGenAI) Workshop

Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models

Tobias Groot, Matias Valdenegro-Toro

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments 8 pages, with appendix. To appear in TrustNLP workshop @ NAACL 2024

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, Baobao Chang

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICLR2024

On Large Visual Language Models for Medical Imaging Analysis: An Empirical Study

Minh-Hao Van, Prateek Verma, Xintao Wu

专题命中 幻觉与鲁棒性 ：visual language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Test-Time Backdoor Attacks on Multimodal Large Language Models

Dong Lu, Tianyu Pang, Chao Du, Qian Liu, Xianjun Yang, Min Lin

专题命中 幻觉与鲁棒性 ：multimodal large language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.LG

On Evaluating Adversarial Robustness of Large Vision-Language Models

Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, Min Lin

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.LG

Comments NeurIPS 2023

Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling

Tsung-Han Wu, Heekyung Lee, Jiaxin Ge, Joseph E. Gonzalez, Trevor Darrell, David M. Chan

机构 * UC Berkeley（加州大学伯克利分校） ； POSTECH 纠错

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract,comments)；分类 cs.CV

Comments Accepted to NeurIPS 2025; Project Page: https://reverse-vlm.github.io

On the Reliability of Vision-Language Models Under Adversarial Frequency-Domain Perturbations

Jordan Vice, Naveed Akhtar, Yansong Gao, Richard Hartley, Ajmal Mian

机构 * University of Western Australia（西澳大学） ； University of Melbourne（墨尔本大学） ； Australian National University（澳大利亚国立大学） 纠错

专题命中 幻觉与鲁棒性 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV

Comments Keywords: Vision-Language Models, Frequency-Domain Perturbations, Adversarial Robustness, Image Authenticity, Reliability

AdaShield: Safeguarding Multimodal Large Language Models from Structure-based Attack via Adaptive Shield Prompting

Yu Wang, Xiaogeng Liu, Yu Li, Muhao Chen, Chaowei Xiao

专题命中 幻觉与鲁棒性 ：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.AI

Comments Multimodal Large Language Models Defense, 25 Pages

MM-PoisonRAG: Disrupting Multimodal RAG with Local and Global Poisoning Attacks

MM-PoisonRAG：通过局部和全局投毒攻击破坏多模态RAG

Hyeonjeong Ha, Qiusi Zhan, Jeonghwan Kim, Dimitrios Bralios, Saikrishna Sanniboina, Nanyun Peng, Kai-Wei Chang, Daniel Kang, Heng Ji

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； University of California Los Angeles（加州大学洛杉矶分校） 纠错

专题命中 幻觉与鲁棒性 ：MLLM(abstract,abstract_cn)；grounding(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI、cs.LG

AI总结 提出MM-PoisonRAG框架，通过局部投毒攻击（LPA）和全局投毒攻击（GPA）两种策略，系统研究多模态检索增强生成（RAG）在知识投毒下的脆弱性，实验表明攻击成功率高达56%且能绕过现有防御。

Comments Code is available at https://github.com/HyeonjeongHa/MM-PoisonRAG

Same Model, Different Weakness: How Language and Modality Reshape the Jailbreak Attack Surface in Frontier MLLMs

相同模型，不同弱点：语言与模态如何重塑前沿多模态大语言模型的越狱攻击面

Casey Ford, Madison Van Doren, Sicheng Jin, Emily Dix

机构 * Appen（Appliance） 纠错

专题命中 幻觉与鲁棒性 ：MLLM(summary_cn,abstract)；multimodal large language model(abstract)

AI总结 通过跨语言多模态红队测试，发现语言和模态通过不同机制影响越狱漏洞，导致安全排名跨语言不守恒，需重新设计评估框架。