多模态大模型

Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models

Songtao Jiang, Tuo Zheng, Yan Zhang, Yeying Jin, Li Yuan, Zuozhu Liu

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion

Xiao Wang, Jiandong Jin, Chenglong Li, Jin Tang, Cheng Zhang, Wei Wang

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by IEEE TCSVT 2024, Camera Ready Version

MultiMath: Bridging Visual and Mathematical Reasoning for Large Language Models

Shuai Peng, Di Fu, Liangcai Gao, Xiuqin Zhong, Hongguang Fu, Zhi Tang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning

Xiaoye Qu, Jiashuo Sun, Wei Wei, Yu Cheng

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 13 pages, 7 tables, 7 figures

Parameter-Efficient Quantized Mixture-of-Experts Meets Vision-Language Instruction Tuning for Semiconductor Electron Micrograph Analysis

Sakhinana Sagar Srinivas, Chidaksh Ravuru, Geethan Sannidhi, Venkataramana Runkana

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Paper published at ICML 2024 Workshop on Foundation Models in the Wild

Foundational Model for Electron Micrograph Analysis: Instruction-Tuning Small-Scale Language-and-Vision Assistant for Enterprise Adoption

Sakhinana Sagar Srinivas, Chidaksh Ravuru, Geethan Sannidhi, Venkataramana Runkana

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Our paper is published at ICML 2024 Workshop ML for Life and Material Science: From Theory to Industry Applications, Vienna, Austria

HYDEN: Hyperbolic Density Representations for Medical Images and Reports

Zhi Qiao, Linbin Han, Xiantong Zhen, Jia-Hong Gao, Zhen Qian

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Adversarial Prompt Tuning for Vision-Language Models

Jiaming Zhang, Xingjun Ma, Xin Wang, Lingyu Qiu, Jiaqi Wang, Yu-Gang Jiang, Jitao Sang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments ECCV 2024

Enhancing Journalism with AI: A Study of Contextualized Image Captioning for News Articles using LLMs and LMMs

Aliki Anagnostopoulou, Thiago Gouvea, Daniel Sonntag

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Teach CLIP to Develop a Number Sense for Ordinal Regression

Yao Du, Qiang Zhai, Weihang Dai, Xiaomeng Li

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted by ECCV 2024

MoExtend: Tuning New Experts for Modality and Task Extension

Shanshan Zhong, Shanghua Gao, Zhongzhan Huang, Wushao Wen, Marinka Zitnik, Pan Zhou

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments ACL 2024 - SRW

Multi-Frame Vision-Language Model for Long-form Reasoning in Driver Behavior Analysis

Hiroshi Takato, Hiroshi Tsutsui, Komei Soda, Hidetaka Kamigaito

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments On-going work

SAT3D: Image-driven Semantic Attribute Transfer in 3D

Zhijun Zhai, Zengmao Wang, Xiaoxiao Long, Kaixuan Zhou, Bo Du

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Journal ref In Proceedings of the 32nd ACM International Conference on Multimedia, 2024

Demonstrating and Reducing Shortcuts in Vision-Language Representation Learning

Maurits Bleeker, Mariya Hendriksen, Andrew Yates, Maarten de Rijke

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments 27 pages, accepted at TMLR

Selective Vision-Language Subspace Projection for Few-shot CLIP

Xingyu Zhu, Beier Zhu, Yi Tan, Shuo Wang, Yanbin Hao, Hanwang Zhang

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.MM

Comments Accepted as an Oral Paper at ACM Multimedia 2024

Enhancing Model Performance: Another Approach to Vision-Language Instruction Tuning

Vedanshu, MM Tripathi, Bhavnesh Jaint

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

A Unified Understanding of Adversarial Vulnerability Regarding Unimodal Models and Vision-Language Pre-training Models

Haonan Zheng, Xinyang Deng, Wen Jiang, Wenrui Li

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 14 pages, 9 figures, published in ACMMM2024(oral)

When Text and Images Don't Mix: Bias-Correcting Language-Image Similarity Scores for Anomaly Detection

Adam Goodge, Bryan Hooi, Wee Siong Ng

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Controllable Contextualized Image Captioning: Directing the Visual Narrative through User-Defined Highlights

Shunqi Mao, Chaoyi Zhang, Hang Su, Hwanjun Song, Igor Shalyminov, Weidong Cai

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments ECCV 2024

GPT Sonograpy: Hand Gesture Decoding from Forearm Ultrasound Images via VLM

Keshav Bimbraw, Ye Wang, Jing Liu, Toshiaki Koike-Akino

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 8 pages, 9 figures

How and where does CLIP process negation?

Vincent Quantmeyer, Pablo Mosteiro, Albert Gatt

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Accepted at the 3rd Workshop on Advances in Language and Vision Research (ALVR 2024)

OTSeg: Multi-prompt Sinkhorn Attention for Zero-Shot Semantic Segmentation

Kwanyoung Kim, Yujin Oh, Jong Chul Ye

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments ECCV 2024; 23 pages, 8 tables, 8 figures; Project Page: https://cubeyoung.github.io/OTSeg_project/

Malicious Path Manipulations via Exploitation of Representation Vulnerabilities of Vision-Language Navigation Systems

Chashi Mahiul Islam, Shaeke Salman, Montasir Shams, Xiuwen Liu, Piyush Kumar

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 8 pages, 5 figures. This paper has been accepted for publication at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2024

The Solution for the 5th GCAIAC Zero-shot Referring Expression Comprehension Challenge

Longfei Huang, Feng Yu, Zhihao Guan, Zhonghua Wan, Yang Yang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output

Pan Zhang, Xiaoyi Dong, Yuhang Zang, Yuhang Cao, Rui Qian, Lin Chen, Qipeng Guo, Haodong Duan, Bin Wang, Linke Ouyang, Songyang Zhang, Wenwei Zhang, Yining Li, Yang Gao, Peng Sun, Xinyue Zhang, Wei Li, Jingwen Li, Wenhai Wang, Hang Yan, Conghui He, Xingcheng Zhang, Kai Chen, Jifeng Dai, Yu Qiao, Dahua Lin, Jiaqi Wang

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments Technical Report. https://github.com/InternLM/InternLM-XComposer

Why do LLaVA Vision-Language Models Reply to Images in English?

Musashi Hinck, Carolin Holtermann, Matthew Lyle Olson, Florian Schneider, Sungduk Yu, Anahita Bhiwandiwalla, Anne Lauscher, Shaoyen Tseng, Vasudev Lal

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Pre-print

Fake News Detection and Manipulation Reasoning via Large Vision-Language Models

Ruihan Jin, Ruibo Fu, Zhengqi Wen, Shuai Zhang, Yukun Liu, Jianhua Tao

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CL、cs.AI

RAVEN: Multitask Retrieval Augmented Vision-Language Learning

Varun Nagaraj Rao, Siddharth Choudhary, Aditya Deshpande, Ravi Kumar Satzoda, Srikar Appalaraju

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Composition Vision-Language Understanding via Segment and Depth Anything Model

Mingxiao Huo, Pengliang Ji, Haotian Lin, Junchen Liu, Yixiao Wang, Yijun Chen

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Constructing Multilingual Visual-Text Datasets Revealing Visual Multilingual Ability of Vision Language Models

Jesse Atuhurra, Iqra Ali, Tatsuya Hiraoka, Hidetaka Kamigaito, Tomoya Iwakura, Taro Watanabe

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL