视觉大模型 / VLM

Can Multimodal LLMs do Visual Temporal Understanding and Reasoning? The answer is No!

Mohamed Fazli Imam, Chenyang Lyu, Alham Fikri Aji

专题命中 视觉问答 ：visual question answering(abstract)；multimodal large language model(abstract)；分类 cs.CV

Comments Our dataset can be found at \url{https://huggingface.co/datasets/fazliimam/temporal-vqa}

Visual Agents as Fast and Slow Thinkers

Guangyan Sun, Mingyu Jin, Zhenting Wang, Cheng-Long Wang, Siqi Ma, Qifan Wang, Tong Geng, Ying Nian Wu, Yongfeng Zhang, Dongfang Liu

专题命中 视觉问答 ：LLaVA(abstract)；visual question answering(abstract)；分类 cs.LG

Comments International Conference on Learning Representations (ICLR 2025)

ReFocus: Visual Editing as a Chain of Thought for Structured Image Understanding

Xingyu Fu, Minqian Liu, Zhengyuan Yang, John Corring, Yijuan Lu, Jianwei Yang, Dan Roth, Dinei Florencio, Cha Zhang

专题命中 视觉问答 ：visual reasoning(abstract)；multimodal large language model(abstract)；分类 cs.CV

Comments Project link: https://zeyofu.github.io/ReFocus/

Cross-Lingual Text-Rich Visual Comprehension: An Information Theory Perspective

Xinmiao Yu, Xiaocheng Feng, Yun Li, Minghui Liao, Ya-Qi Yu, Xiachong Feng, Weihong Zhong, Ruihan Chen, Mengkang Hu, Jihao Wu, Dandan Tu, Duyu Tang, Bing Qin

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Beyond End-to-End VLMs: Leveraging Intermediate Text Representations for Superior Flowchart Understanding

Junyi Ye, Ankan Dash, Wenpeng Yin, Guiling Wang

专题命中 视觉问答 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

ViUniT: Visual Unit Tests for More Robust Visual Programming

Artemis Panagopoulou, Honglu Zhou, Silvio Savarese, Caiming Xiong, Chris Callison-Burch, Mark Yatskar, Juan Carlos Niebles

专题命中 视觉问答 ：visual reasoning(abstract)；visual question answering(abstract)；分类 cs.CV

Towards Foundation Models for 3D Vision: How Close Are We?

Yiming Zuo, Karhan Kayan, Maggie Wang, Kevin Jeon, Jia Deng, Thomas L. Griffiths

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Accepted to 3DV 2025. Update 12/09/24: Change the benchmark name to UniQA-3D, add link to code

Awaker2.5-VL: Stably Scaling MLLMs with Parameter-Efficient Mixture of Experts

Jinqiang Long, Yanqi Dai, Guoxing Yang, Hongpeng Lin, Nanyi Fei, Yizhao Gao, Zhiwu Lu

专题命中 视觉问答 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Select2Plan: Training-Free ICL-Based Planning through VQA and Memory Retrieval

Davide Buoso, Luke Robinson, Giuseppe Averta, Philip Torr, Tim Franzmeyer, Daniele De Martini

专题命中 视觉问答 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

SimpsonsVQA: Enhancing Inquiry-Based Learning with a Tailored Dataset

Ngoc Dung Huynh, Mohamed Reda Bouadjenek, Sunil Aryal, Imran Razzak, Hakim Hacid

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Are VLMs Really Blind

Ayush Singh, Mansi Gupta, Shivank Garg

专题命中 视觉问答 ：vision language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments 2 pages, 1 figure

GMAI-MMBench: A Comprehensive Multimodal Evaluation Benchmark Towards General Medical AI

Pengcheng Chen, Jin Ye, Guoan Wang, Yanjun Li, Zhongying Deng, Wei Li, Tianbin Li, Haodong Duan, Ziyan Huang, Yanzhou Su, Benyou Wang, Shaoting Zhang, Bin Fu, Jianfei Cai, Bohan Zhuang, Eric J Seibel, Junjun He, Yu Qiao

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments GitHub: https://github.com/uni-medical/GMAI-MMBench Hugging face: https://huggingface.co/datasets/OpenGVLab/GMAI-MMBench

OMCAT: Omni Context Aware Transformer

Arushi Goel, Karan Sapra, Matthieu Le, Rafael Valle, Andrew Tao, Bryan Catanzaro

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV

Comments Demo page: https://om-cat.github.io

Difficult Task Yes but Simple Task No: Unveiling the Laziness in Multimodal LLMs

Sihang Zhao, Youliang Yuan, Xiaoying Tang, Pinjia He

专题命中 视觉问答 ：LLaVA(abstract)；multimodal large language model(abstract)；分类 cs.AI

Comments EMNLP 2024 Findings

NEVLP: Noise-Robust Framework for Efficient Vision-Language Pre-training

Yiyi Tao, Zhuoyue Wang, Hang Zhang, Lun Wang

专题命中 视觉问答 ：vision language model(abstract)；visual question answering(abstract)；分类 cs.CV

MOKA: Open-World Robotic Manipulation through Mark-Based Visual Prompting

Fangchen Liu, Kuan Fang, Pieter Abbeel, Sergey Levine

专题命中 视觉问答 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

PropTest: Automatic Property Testing for Improved Visual Programming

Jaywon Koo, Ziyan Yang, Paola Cascante-Bonilla, Baishakhi Ray, Vicente Ordonez

专题命中 视觉问答 ：visual reasoning(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Project Page: https://jaywonkoo17.github.io/PropTest/

How Easy is It to Fool Your Multimodal LLMs? An Empirical Analysis on Deceptive Prompts

Yusu Qian, Haotian Zhang, Yinfei Yang, Zhe Gan

专题命中 视觉问答 ：LLaVA(abstract)；multimodal large language model(abstract)；分类 cs.CV

GPT-4V Explorations: Mining Autonomous Driving

Zixuan Li

专题命中 视觉问答 ：visual language model(abstract)；visual question answering(abstract)；分类 cs.CV

Instruction Makes a Difference

Tosin Adewumi, Nudrat Habib, Lama Alkhaled, Elisa Barney

专题命中 视觉问答 ：LLaVA(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Accepted at the 16th IAPR International Workshop On Document Analysis Systems (DAS)

Understanding Information Storage and Transfer in Multi-modal Large Language Models

Samyadeep Basu, Martin Grayson, Cecily Morrison, Besmira Nushi, Soheil Feizi, Daniela Massiceti

专题命中 视觉问答 ：LLaVA(abstract)；visual question answering(abstract)；分类 cs.CV

Comments 20 pages

Hawk: Learning to Understand Open-World Video Anomalies

Jiaqi Tang, Hao Lu, Ruizheng Wu, Xiaogang Xu, Ke Ma, Cheng Fang, Bin Guo, Jiangbo Lu, Qifeng Chen, Ying-Cong Chen

专题命中 视觉问答 ：VLM(abstract)；visual language model(abstract)；分类 cs.CV

OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLM

Yutao Hu, Tianbin Li, Quanfeng Lu, Wenqi Shao, Junjun He, Yu Qiao, Ping Luo

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Language-Informed Visual Concept Learning

Sharon Lee, Yunzhi Zhang, Shangzhe Wu, Jiajun Wu

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments ICLR 2024. The first two authors contributed equally and are alphabetically ordered. Project page: https://ai.stanford.edu/~yzzhang/projects/concept-axes/

WoLF: Wide-scope Large Language Model Framework for CXR Understanding

Seil Kang, Donghyun Kim, Junhyeok Kim, Hyo Kyung Lee, Seong Jae Hwang

专题命中 视觉问答 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.AI

Comments 11 pages main paper, 2 pages supplementary

From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Models

Dongsheng Jiang, Yuchen Liu, Songlin Liu, Jin'e Zhao, Hao Zhang, Zhen Gao, Xiaopeng Zhang, Jin Li, Hongkai Xiong

专题命中 视觉问答 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV

Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applications

Muhammad Arslan Manzoor, Sarah Albarri, Ziting Xian, Zaiqiao Meng, Preslav Nakov, Shangsong Liang

专题命中 视觉问答 ：visual reasoning(abstract)；visual question answering(abstract)；分类 cs.AI

SceMQA: A Scientific College Entrance Level Multimodal Question Answering Benchmark

Zhenwen Liang, Kehan Guo, Gang Liu, Taicheng Guo, Yujun Zhou, Tianyu Yang, Jiajun Jiao, Renjie Pi, Jipeng Zhang, Xiangliang Zhang

专题命中 视觉问答 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI

Comments Work in progress

LiDAR-LLM: Exploring the Potential of Large Language Models for 3D LiDAR Understanding

Senqiao Yang, Jiaming Liu, Ray Zhang, Mingjie Pan, Zoey Guo, Xiaoqi Li, Zehui Chen, Peng Gao, Yandong Guo, Shanghang Zhang

专题命中 视觉问答 ：grounding(abstract)；multimodal large language model(abstract)；分类 cs.CV

NuScenes-MQA: Integrated Evaluation of Captions and QA for Autonomous Driving Datasets using Markup Annotations

Yuichi Inoue, Yuki Yada, Kotaro Tanahashi, Yu Yamaguchi

专题命中 视觉问答 ：vision language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Accepted at LLVM-AD Workshop @ WACV 2024