视觉大模型 / VLM

EHRXQA: A Multi-Modal Question Answering Dataset for Electronic Health Records with Chest X-ray Images

Seongsu Bae, Daeun Kyung, Jaehee Ryu, Eunbyeol Cho, Gyubok Lee, Sunjun Kweon, Jungwoo Oh, Lei Ji, Eric I-Chao Chang, Tackeun Kim, Edward Choi

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Accepted at NeurIPS 2023 Datasets and Benchmarks Track (10 pages for main text, 4 pages for references, 39 pages for supplementary materials)

MedAlign: A Clinician-Generated Dataset for Instruction Following with Electronic Medical Records

Scott L. Fleming, Alejandro Lozano, William J. Haberkorn, Jenelle A. Jindal, Eduardo P. Reis, Rahul Thapa, Louis Blankemeier, Julian Z. Genkins, Ethan Steinberg, Ashwin Nayak, Birju S. Patel, Chia-Chun Chiang, Alison Callahan, Zepeng Huo, Sergios Gatidis, Scott J. Adams, Oluseyi Fayanju, Shreya J. Shah, Thomas Savage, Ethan Goh, Akshay S. Chaudhari, Nima Aghaeepour, Christopher Sharp, Michael A. Pfeffer, Percy Liang, Jonathan H. Chen, Keith E. Morse, Emma P. Brunskill, Jason A. Fries, Nigam H. Shah

专题命中 视觉问答 ：grounding(abstract)；分类 cs.AI、cs.LG

Unleashing the Potential of Large Language Model: Zero-shot VQA for Flood Disaster Scenario

Yimin Sun, Chao Wang, Yan Peng

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments accepted by The 4th International Conference on Artificial Intelligence and Computer Engineering

The curse of language biases in remote sensing VQA: the role of spatial attributes, language diversity, and the need for clear evaluation

Christel Chappuis, Eliot Walt, Vincent Mendez, Sylvain Lobry, Bertrand Le Saux, Devis Tuia

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Zero-shot Translation of Attention Patterns in VQA Models to Natural Language

Leonard Salewski, A. Sophia Koepke, Hendrik P. A. Lensch, Zeynep Akata

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Published in GCPR 2023

Dynamic Task and Weight Prioritization Curriculum Learning for Multimodal Imagery

Huseyin Fuat Alsan, Taner Arsan

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

AVIS: Autonomous Visual Information Seeking with Large Language Model Agent

Ziniu Hu, Ahmet Iscen, Chen Sun, Kai-Wei Chang, Yizhou Sun, David A Ross, Cordelia Schmid, Alireza Fathi

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Published on NeurIPS 2023

Image Manipulation via Multi-Hop Instructions -- A New Dataset and Weakly-Supervised Neuro-Symbolic Approach

Harman Singh, Poorva Garg, Mohit Gupta, Kevin Shah, Ashish Goswami, Satyam Modi, Arnab Kumar Mondal, Dinesh Khandelwal, Dinesh Garg, Parag Singla

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments EMNLP 2023 (long paper, main conference)

tagE: Enabling an Embodied Agent to Understand Human Instructions

Chayan Sarkar, Avik Mitra, Pradip Pramanick, Tapas Nayak

专题命中 视觉问答 ：grounding(abstract)；分类 cs.AI、cs.LG

Comments Accepted in EMNLP Findings 2023

Can Pre-trained Vision and Language Models Answer Visual Information-Seeking Questions?

Yang Chen, Hexiang Hu, Yi Luan, Haitian Sun, Soravit Changpinyo, Alan Ritter, Ming-Wei Chang

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments EMNLP 2023 (main conference); Our dataset and evaluation is available at https://open-vision-language.github.io/infoseek/

Ground Truth Evaluation of Neural Network Explanations with CLEVR-XAI

Leila Arras, Ahmed Osman, Wojciech Samek

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.LG

Comments 37 pages, 9 tables, 2 figures (plus appendix 14 pages)

Rad-ReStruct: A Novel VQA Benchmark and Method for Structured Radiology Reporting

Chantal Pellegrini, Matthias Keicher, Ege Özsoy, Nassir Navab

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments accepted at MICCAI 2023

Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP

Vedant Palit, Rohan Pandey, Aryaman Arora, Paul Pu Liang

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Final version for 5th Workshop on Closing the Loop Between Vision and Language (CLVL) @ ICCV 2023. 4 pages, 5 figures

TeCH: Text-guided Reconstruction of Lifelike Clothed Humans

Yangyi Huang, Hongwei Yi, Yuliang Xiu, Tingting Liao, Jiaxiang Tang, Deng Cai, Justus Thies

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Project: https://huangyangyi.github.io/TeCH, Code: https://github.com/huangyangyi/TeCH

CAT-ViL: Co-Attention Gated Vision-Language Embedding for Visual Question Localized-Answering in Robotic Surgery

Long Bai, Mobarakol Islam, Hongliang Ren

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments To appear in MICCAI 2023. Code availability: https://github.com/longbai1006/CAT-ViL

Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Images

Nitzan Bitton-Guetta, Yonatan Bitton, Jack Hessel, Ludwig Schmidt, Yuval Elovici, Gabriel Stanovsky, Roy Schwartz

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Accepted to ICCV 2023. Website: whoops-benchmark.github.io

Foundation Model is Efficient Multimodal Multitask Model Selector

Fanqing Meng, Wenqi Shao, Zhanglin Peng, Chonghe Jiang, Kaipeng Zhang, Yu Qiao, Ping Luo

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.AI、cs.LG

Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner

Zikang Liu, Sihan Chen, Longteng Guo, Handong Li, Xingjian He, Jing Liu

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 12 pages. Accepted by ACM MM '23

A reinforcement learning approach for VQA validation: an application to diabetic macular edema grading

Tatiana Fountoukidou, Raphael Sznitman

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 16 pages (+ 23 pages supplementary material)

Journal ref Medical image analysis 87 (2023): 102822

Compositionality as Lexical Symmetry

Ekin Akyürek, Jacob Andreas

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.LG

Comments ACL2023 Final Version

Switch-BERT: Learning to Model Multimodal Interactions by Switching Attention and Input

Qingpei Guo, Kaisheng Yao, Wei Chu

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ECCV2022

Q2ATransformer: Improving Medical VQA via an Answer Querying Decoder

Yunyi Liu, Zhanyu Wang, Dong Xu, Luping Zhou

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Using Visual Cropping to Enhance Fine-Detail Question Answering of BLIP-Family Models

Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, Filip Ilievski

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments 16 pages, 5 figures, 7 tables

i-Code Studio: A Configurable and Composable Framework for Integrative AI

Yuwei Fang, Mahmoud Khademi, Chenguang Zhu, Ziyi Yang, Reid Pryzant, Yichong Xu, Yao Qian, Takuya Yoshioka, Lu Yuan, Michael Zeng, Xuedong Huang

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Combo of Thinking and Observing for Outside-Knowledge VQA

Qingyi Si, Yuchen Mo, Zheng Lin, Huishan Ji, Weiping Wang

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments ACL-23, Main Conference

Multimodal Federated Learning via Contrastive Representation Ensemble

Qiying Yu, Yang Liu, Yimu Wang, Ke Xu, Jingjing Liu

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.AI、cs.LG

Comments ICLR 2023, update

Locate Then Generate: Bridging Vision and Language with Bounding Box for Scene-Text VQA

Yongxin Zhu, Zhen Liu, Yukang Liang, Xin Li, Hao Liu, Changcun Bao, Linli Xu

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments accepted in AAAI 2023

VQA with Cascade of Self- and Co-Attention Blocks

Aakansha Mishra, Ashish Anand, Prithwijit Guha

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

MQA: Answering the Question via Robotic Manipulation

Yuhong Deng, Di Guo, Xiaofeng Guo, Naifu Zhang, Huaping Liu, Fuchun Sun

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments has been accepted by Robotics: Science and Systems 2021

Journal ref Robotics: Science and System (RSS2021)

BinaryVQA: A Versatile Test Set to Evaluate the Out-of-Distribution Generalization of VQA Models

Ali Borji

专题命中 视觉问答 ：visual question answering(abstract)；分类 cs.CV、cs.AI