多模态大模型

Benchmarking Multi-Image Understanding in Vision and Language Models: Perception, Knowledge, Reasoning, and Multi-Hop Reasoning

Bingchen Zhao, Yongshuo Zong, Letian Zhang, Timothy Hospedales

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments First three authors contributed equally. Dataset: https://huggingface.co/datasets/VLLMs/MIRB

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

Yujie Lu, Dongfu Jiang, Wenhu Chen, William Yang Wang, Yejin Choi, Bill Yuchen Lin

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments link: https://hf.co/spaces/WildVision/vision-arena

Exploiting LMM-based knowledge for image classification tasks

Maria Tzelepi, Vasileios Mezaris

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted for publication, 25th Int. Conf. on Engineering Applications of Neural Networks (EANN/EAAAI 2024), Corfu, Greece, June 2024. This is the "submitted manuscript"

Text-only Synthesis for Image Captioning

Qing Zhou, Junlin Huang, Qiang Li, Junyu Gao, Qi Wang

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Revisiting the Role of Language Priors in Vision-Language Models

Zhiqiu Lin, Xinyue Chen, Deepak Pathak, Pengchuan Zhang, Deva Ramanan

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Published at ICML 2024. Website: https://linzhiqiu.github.io/papers/visual_gpt_score/

Skip \n: A Simple Method to Reduce Hallucination in Large Vision-Language Models

Zongbo Han, Zechen Bai, Haiyang Mei, Qianli Xu, Changqing Zhang, Mike Zheng Shou

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

CLIP as RNN: Segment Countless Visual Concepts without Training Endeavor

Shuyang Sun, Runjia Li, Philip Torr, Xiuye Gu, Siyang Li

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments To appear in CVPR 2024. Project page: https://torrvision.com/clip_as_rnn/

Harnessing GPT-4V(ision) for Insurance: A Preliminary Exploration

Chenwei Lin, Hanjia Lyu, Jiebo Luo, Xian Xu

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Concept-based Analysis of Neural Networks via Vision-Language Models

Ravi Mangal, Nina Narodytska, Divya Gopinath, Boyue Caroline Hu, Anirban Roy, Susmit Jha, Corina Pasareanu

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

ALOHa: A New Measure for Hallucination in Captioning Models

Suzanne Petryk, David M. Chan, Anish Kachinthaya, Haodi Zou, John Canny, Joseph E. Gonzalez, Trevor Darrell

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments To appear at NAACL 2024

OSCaR: Object State Captioning and State Change Representation

Nguyen Nguyen, Jing Bi, Ali Vosoughi, Yapeng Tian, Pooyan Fazli, Chenliang Xu

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NAACL 2024

MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metric

Haokun Lin, Haoli Bai, Zhili Liu, Lu Hou, Muyi Sun, Linqi Song, Ying Wei, Zhenan Sun

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 18 pages, 8 figures, Published in CVPR2024

Journal ref In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2024

Contrastive Region Guidance: Improving Grounding in Vision-Language Models without Training

David Wan, Jaemin Cho, Elias Stengel-Eskin, Mohit Bansal

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project website: https://contrastive-region-guidance.github.io/

Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answering

Junnan Dong, Qinggang Zhang, Huachi Zhou, Daochen Zha, Pai Zheng, Xiao Huang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 8 pages,3 figures and 1 page appendix; The processed graphs and codes will be avalibale

MouSi: Poly-Visual-Expert Vision-Language Models

Xiaoran Fan, Tao Ji, Changhao Jiang, Shuo Li, Senjie Jin, Sirui Song, Junke Wang, Boyang Hong, Lu Chen, Guodong Zheng, Ming Zhang, Caishuang Huang, Rui Zheng, Zhiheng Xi, Yuhao Zhou, Shihan Dou, Junjie Ye, Hang Yan, Tao Gui, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Zuxuan Wu, Yu-Gang Jiang

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Models

Yi-Lin Sung, Jaehong Yoon, Mohit Bansal

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICLR 2024 (project page: https://ecoflap.github.io/)

NExT-Chat: An LMM for Chat, Detection and Segmentation

Ao Zhang, Yuan Yao, Wei Ji, Zhiyuan Liu, Tat-Seng Chua

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Technical Report (https://next-chatv.github.io/)

Alpha-CLIP: A CLIP Model Focusing on Wherever You Want

Zeyi Sun, Ye Fang, Tong Wu, Pan Zhang, Yuhang Zang, Shu Kong, Yuanjun Xiong, Dahua Lin, Jiaqi Wang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments project page: https://aleafy.github.io/alpha-clip code: https://github.com/SunzeY/AlphaCLIP

Visual Instruction Tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2023 Oral; project page: https://llava-vl.github.io/

Localized Symbolic Knowledge Distillation for Visual Commonsense Models

Jae Sung Park, Jack Hessel, Khyathi Raghavi Chandu, Paul Pu Liang, Ximing Lu, Peter West, Youngjae Yu, Qiuyuan Huang, Jianfeng Gao, Ali Farhadi, Yejin Choi

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Neurips 2023

Representing visual classification as a linear combination of words

Shobhit Agarwal, Yevgeniy R. Semenov, William Lotter

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments To be published in the Proceedings of the 3rd Machine Learning for Health symposium, Proceedings of Machine Learning Research (PMLR)

FuseCap: Leveraging Large Language Models for Enriched Fused Image Captions

Noam Rotstein, David Bensaid, Shaked Brody, Roy Ganz, Ron Kimmel

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

On Evaluating Adversarial Robustness of Large Vision-Language Models

Yunqing Zhao, Tianyu Pang, Chao Du, Xiao Yang, Chongxuan Li, Ngai-Man Cheung, Min Lin

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments NeurIPS 2023

Evaluating Object Hallucination in Large Vision-Language Models

Yifan Li, Yifan Du, Kun Zhou, Jinpeng Wang, Wayne Xin Zhao, Ji-Rong Wen

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted to EMNLP 2023

Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIP

Vedant Palit, Rohan Pandey, Aryaman Arora, Paul Pu Liang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Final version for 5th Workshop on Closing the Loop Between Vision and Language (CLVL) @ ICCV 2023. 4 pages, 5 figures

Vision Meets Definitions: Unsupervised Visual Word Sense Disambiguation Incorporating Gloss Information

Sunjae Kwon, Rishabh Garodia, Minhwa Lee, Zhichao Yang, Hong Yu

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2023, https://aclanthology.org/2023.acl-long.88

Using Visual Cropping to Enhance Fine-Detail Question Answering of BLIP-Family Models

Jiarui Zhang, Mahyar Khayatkhoei, Prateek Chhikara, Filip Ilievski

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 16 pages, 5 figures, 7 tables

Zero-shot Referring Image Segmentation with Global-Local Context Features

Seonghoon Yu, Paul Hongsuck Seo, Jeany Son

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2023

See Your Heart: Psychological states Interpretation through Visual Creations

Likun Yang, Xiaokun Feng, Xiaotang Chen, Shiyu Zhang, Kaiqi Huang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

PMC-CLIP: Contrastive Language-Image Pre-training using Biomedical Documents

Weixiong Lin, Ziheng Zhao, Xiaoman Zhang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, Weidi Xie

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments 10 pages, 3 figures