多模态大模型

How Does Vision-Language Adaptation Impact the Safety of Vision Language Models?

Seongyun Lee, Geewook Kim, Jiyeon Kim, Hyunji Lee, Hoyeon Chang, Sue Hyun Park, Minjoon Seo

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Work in Progress

TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives

Maitreya Patel, Abhiram Kusumba, Sheng Cheng, Changhoon Kim, Tejas Gokhale, Chitta Baral, Yezhou Yang

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted at: NeurIPS 2024 | Project Page: https://tripletclip.github.io

Vision-Aware Text Features in Referring Image Segmentation: From Object Understanding to Context Understanding

Hai Nguyen-Truong, E-Ro Nguyen, Tuan-Anh Vu, Minh-Triet Tran, Binh-Son Hua, Sai-Kit Yeung

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments This paper is accepted in WACV 2025

OmniJARVIS: Unified Vision-Language-Action Tokenization Enables Open-World Instruction Following Agents

Zihao Wang, Shaofei Cai, Zhancun Mu, Haowei Lin, Ceyao Zhang, Xuejie Liu, Qing Li, Anji Liu, Xiaojian Ma, Yitao Liang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments accepted on NeurIPS 2024

Natural Language Inference Improves Compositionality in Vision-Language Models

Paola Cascante-Bonilla, Yu Hou, Yang Trista Cao, Hal Daumé, Rachel Rudinger

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments Project page: https://cece-vlm.github.io/

MMDocBench: Benchmarking Large Vision-Language Models for Fine-Grained Visual Document Understanding

Fengbin Zhu, Ziyang Liu, Xiang Yao Ng, Haohui Wu, Wenjie Wang, Fuli Feng, Chao Wang, Huanbo Luan, Tat Seng Chua

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Under review

A Unified Debiasing Approach for Vision-Language Models across Modalities and Tasks

Hoin Jung, Taeuk Jang, Xiaoqian Wang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2024 (Spotlight), the Thirty-Eighth Annual Conference on Neural Information Processing Systems

Effectiveness Assessment of Recent Large Vision-Language Models

Yao Jiang, Xinyu Yan, Ge-Peng Ji, Keren Fu, Meijun Sun, Huan Xiong, Deng-Ping Fan, Fahad Shahbaz Khan

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by Visual Intelligence

Journal ref Visual Intelligence, 2024, Vol. 2, article no. 17

IPO: Interpretable Prompt Optimization for Vision-Language Models

Yingjun Du, Wenfang Sun, Cees G. M. Snoek

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Accepted by NeurIPS 2024

VLFeedback: A Large-Scale AI Feedback Dataset for Large Vision-Language Models Alignment

Lei Li, Zhihui Xie, Mukai Li, Shunian Chen, Peiyi Wang, Liang Chen, Yazheng Yang, Benyou Wang, Lingpeng Kong, Qi Liu

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments EMNLP 2024 Main Conference camera-ready version (fixed small typos). This article supersedes arXiv:2312.10665

VL-GLUE: A Suite of Fundamental yet Challenging Visuo-Linguistic Reasoning Tasks

Shailaja Keyur Sampat, Mutsumi Nakamura, Shankar Kailas, Kartik Aggarwal, Mandy Zhou, Yezhou Yang, Chitta Baral

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments 18 pages, 7 figures

GeoCoder: Solving Geometry Problems by Generating Modular Code through Vision-Language Models

Aditya Sharma, Aman Dalmia, Mehran Kazemi, Amal Zouaq, Christopher J. Pal

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Surgical-LLaVA: Toward Surgical Scenario Understanding via Large Language and Vision Models

Juseong Jin, Chang Wook Jeong

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2024 AIM-FM Workshop

Unraveling Cross-Modality Knowledge Conflicts in Large Vision-Language Models

Tinghui Zhu, Qin Liu, Fei Wang, Zhengzhong Tu, Muhao Chen

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Website: https://darthzhu.github.io/cross-modality-knowledge-conflict/

CUS3D :CLIP-based Unsupervised 3D Segmentation via Object-level Denoise

Fuyang Yu, Runze Tian, Zhen Wang, Xiaochuan Wang, Xiaohui Liang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM

Comments 6 pages,3 figures

How Does Diverse Interpretability of Textual Prompts Impact Medical Vision-Language Zero-Shot Tasks?

Sicheng Wang, Che Liu, Rossella Arcucci

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted by NeurIPS'24 Advancements In Medical Foundation Models: Explainability, Robustness, Security, and Beyond Workshop

Break the Visual Perception: Adversarial Attacks Targeting Encoded Visual Tokens of Large Vision-Language Models

Yubo Wang, Chaohu Liu, Yanqiu Qu, Haoyu Cao, Deqiang Jiang, Linli Xu

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted to ACMMM 2024

TuneVLSeg: Prompt Tuning Benchmark for Vision-Language Segmentation Models

Rabin Adhikari, Safal Thapaliya, Manish Dhakal, Bishesh Khanal

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Accepted at ACCV 2024 (oral presentation)

ESREAL: Exploiting Semantic Reconstruction to Mitigate Hallucinations in Vision-Language Models

Minchan Kim, Minyeong Kim, Junik Bae, Suhwan Choi, Sungkyung Kim, Buru Chang

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments ECCV 2024

The age of spiritual machines: Language quietus induces synthetic altered states of consciousness in artificial intelligence

Jeremy I Skipper, Joanna Kuc, Greg Cooper, Christopher Timmermann

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 8 Figures

Shaking Up VLMs: Comparing Transformers and Structured State Space Models for Vision & Language Modeling

Georgios Pantazopoulos, Malvina Nikandrou, Alessandro Suglia, Oliver Lemon, Arash Eshghi

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding

Fan Yuan, Chi Qin, Xiaogang Xu, Piji Li

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Accepted at Main Conference of EMNLP 2024

The Hard Positive Truth about Vision-Language Compositionality

Amita Kamath, Cheng-Yu Hsieh, Kai-Wei Chang, Ranjay Krishna

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments ECCV 2024

P4Q: Learning to Prompt for Quantization in Visual-language Models

Huixin Sun, Runqi Wang, Yanjing Li, Xianbin Cao, Xiaolong Jiang, Yao Hu, Baochang Zhang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Can CLIP Count Stars? An Empirical Study on Quantity Bias in CLIP

Zeliang Zhang, Zhuo Liu, Mingqian Feng, Chenliang Xu

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Short paper. Accepted by the Findings of EMNLP 2024

Pseudo-Prompt Generating in Pre-trained Vision-Language Models for Multi-Label Medical Image Classification

Yaoqin Ye, Junjie Zhang, Hongwei Shi

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted by PRCV 2024

ComAlign: Compositional Alignment in Vision-Language Models

Ali Abdollah, Amirmohammad Izadi, Armin Saghafian, Reza Vahidimajd, Mohammad Mozafari, Amirreza Mirzaei, Mohammadmahdi Samiei, Mahdieh Soleymani Baghshah

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.MM

Securing Vision-Language Models with a Robust Encoder Against Jailbreak and Adversarial Attacks

Md Zarif Hossain, Ahmed Imteaj

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questions

Yudong Zhang, Ruobing Xie, Jiansheng Chen, Xingwu Sun, Yu Wang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ACM Multimedia 2024 BNI track (Oral)

Semantically Controllable Augmentations for Generalizable Robot Learning

Zoey Chen, Zhao Mandi, Homanga Bharadhwaj, Mohit Sharma, Shuran Song, Abhishek Gupta, Vikash Kumar

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted for publication by IJRR. First 3 authors contributed equally. Last 3 authors advised equally