多模态大模型 - arXivDaily 专题

2407.00020 2024-07-02 cs.CV cs.AI cs.CL cs.IT cs.LG math.IT 82%

Visual Language Model based Cross-modal Semantic Communication Systems

Feibo Jiang, Chuanguo Tang, Li Dong, Kezhi Wang, Kun Yang, Cunhua Pan

专题命中图文多模态：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 12 pages, 10 figures

2406.19736 2024-07-01 cs.CV cs.AI cs.CL cs.LG 82%

MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment

Jihao Liu, Xin Huang, Jinliang Zheng, Boxiao Liu, Jia Wang, Osamu Yoshie, Yu Liu, Hongsheng Li

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Dataset and models are available at https://github.com/jihaonew/MM-Instruct

2406.11048 2024-06-18 cs.LG cs.DC 82%

Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality

Liwei Che, Jiaqi Wang, Xinyue Liu, Fenglong Ma

专题命中图文多模态：multi-modal(title,abstract)；image-text(abstract)

Comments Accepted by ECML-PKDD 2024

2406.06004 2024-06-11 cs.CV cs.AI cs.CL 82%

FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model

Yebin Lee, Imseong Park, Myungjoo Kang

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at ACL (Main) 2024

2405.01583 2024-05-06 cs.CL cs.AI cs.CV cs.LG 82%

MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning

Nadia Saeed

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 7 pages, 3 figures, Clinical NLP 2024 workshop proceedings in Shared Task

2312.00784 2024-04-30 cs.CV cs.AI cs.CL cs.LG 82%

ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

Mu Cai, Haotian Liu, Dennis Park, Siva Karthik Mustikovela, Gregory P. Meyer, Yuning Chai, Yong Jae Lee

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to CVPR2024. Project page: https://vip-llava.github.io/

2404.15653 2024-04-25 cs.CV cs.AI cs.CL cs.LG 82%

CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data

Sachin Mehta, Maxwell Horton, Fartash Faghri, Mohammad Hossein Sekhavat, Mahyar Najibi, Mehrdad Farajtabar, Oncel Tuzel, Mohammad Rastegari

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2404.15532 2024-04-25 cs.HC cs.AI cs.CL cs.CV cs.MA 82%

BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysis

Shuhang Lin, Wenyue Hua, Lingyao Li, Che-Jui Chang, Lizhou Fan, Jianchao Ji, Hang Hua, Mingyu Jin, Jiebo Luo, Yongfeng Zhang

专题命中图文多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 26 pages, 14 figures The data and code for this project are accessible at https://github.com/agiresearch/battleagent

2309.07915 2024-03-21 cs.CL cs.AI cs.CV 82%

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

Haozhe Zhao, Zefan Cai, Shuzheng Si, Xiaojian Ma, Kaikai An, Liang Chen, Zixuan Liu, Sheng Wang, Wenjuan Han, Baobao Chang

专题命中图文多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by ICLR2024

2403.06835 2024-03-12 cs.CV cs.AI cs.CL 82%

Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Prompting

Wenting Chen, Pengyu Wang, Hui Ren, Lichao Sun, Quanzheng Li, Yixuan Yuan, Xiang Li

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 5 figures

2402.18091 2024-02-29 cs.CV cs.AI cs.CL 82%

Polos: Multimodal Metric Learning from Human Feedback for Image Captioning

Yuiga Wada, Kanta Kaneda, Daichi Saito, Komei Sugiura

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2024

2310.03211 2023-11-29 cs.CL cs.AI cs.CV 82%

On the Performance of Multimodal Language Models

Utsav Garg, Erhan Bas

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2310.18369 2023-10-31 cs.CL cs.AI cs.CV 82%

Apollo: Zero-shot MultiModal Reasoning with Multiple Experts

Daniela Ben-David, Tzuf Paz-Argaman, Reut Tsarfaty

专题命中图文多模态：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments GitHub: https://github.com/danielabd/Apollo-Cap

2302.05608 2023-10-09 cs.CV cs.AI cs.CL cs.LG 82%

Differentiable Outlier Detection Enable Robust Deep Multimodal Analysis

Zhu Wang, Sourav Medya, Sathya N. Ravi

专题命中图文多模态：multimodal(title)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

2308.15226 2023-08-30 cs.CV cs.AI cs.CL 82%

CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation

Devaansh Gupta, Siddhant Kharbanda, Jiawei Zhou, Wanhua Li, Hanspeter Pfister, Donglai Wei

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 15 pages, 9 figures, to be published In Proceedings of International Conference of Computer Vision(ICCV), 2023

2209.06997 2022-09-16 cs.LG cs.CR 82%

M^4I: Multi-modal Models Membership Inference

Pingyi Hu, Zihan Wang, Ruoxi Sun, Hu Wang, Minhui Xue

专题命中图文多模态：multi-modal(title,abstract)；multimodal(abstract)

Comments Accepted to NeurIPS 2022

2204.00598 2022-05-30 cs.CV cs.AI cs.CL cs.LG 82%

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

Andy Zeng, Maria Attarian, Brian Ichter, Krzysztof Choromanski, Adrian Wong, Stefan Welker, Federico Tombari, Aveek Purohit, Michael Ryoo, Vikas Sindhwani, Johnny Lee, Vincent Vanhoucke, Pete Florence

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments https://socraticmodels.github.io/

2204.07955 2022-04-22 cs.CV cs.CL cs.MM 82%

Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Yan Ling, Jianfei Yu, Rui Xia

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted by ACL 2022 (long paper)

2201.11517 2022-01-28 q-bio.NC cs.LG cs.NE 82%

Multimodal neural networks better explain multivoxel patterns in the hippocampus

Bhavin Choksi, Milad Mozafari, Rufin VanRullen, Leila Reddy

专题命中图文多模态：multimodal(title,abstract)；multi-modal(abstract)

Comments Oral at SVRHM Workshop (NeurIPS 2021)

2010.08725 2020-10-20 cs.CL cs.AI cs.CV 82%

A Corpus for English-Japanese Multimodal Neural Machine Translation with Comparable Sentences

Andrew Merritt, Chenhui Chu, Yuki Arase

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2009.11278 2020-09-24 cs.CV cs.AI cs.CL cs.LG 82%

X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

Jaemin Cho, Jiasen Lu, Dustin Schwenk, Hannaneh Hajishirzi, Aniruddha Kembhavi

专题命中图文多模态：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2020

1905.12260 2020-07-02 cs.CL cs.AI cs.CV cs.LG 82%

Learning Multilingual Word Embeddings Using Image-Text Data

Karan Singhal, Karthik Raman, Balder ten Cate

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2004.14973 2020-05-04 cs.CV cs.AI cs.CL cs.LG 82%

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, Dhruv Batra

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.AI

1907.04378 2019-07-11 cs.CV cs.CL cs.LG eess.AS eess.IV 82%

M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention

Shuang Ma, Daniel McDuff, Yale Song

专题命中图文多模态：multi-modal(title)；cross-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

1906.06892 2019-06-18 cs.CV cs.CL cs.LG cs.MM 82%

ParNet: Position-aware Aggregated Relation Network for Image-Text matching

Yaxian Xia, Lun Huang, Wenmin Wang, Xiaoyong Wei, Wenmin Wang

专题命中图文多模态：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments 9 pages, 3 figures

1811.10092 2019-04-09 cs.CV cs.AI cs.CL cs.RO 82%

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Xin Wang, Qiuyuan Huang, Asli Celikyilmaz, Jianfeng Gao, Dinghan Shen, Yuan-Fang Wang, William Yang Wang, Lei Zhang

专题命中图文多模态：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2019 Oral

1712.03404 2017-12-12 cs.IR 82%

Semi-supervised Multimodal Hashing

Dayong Tian, Maoguo Gong, Deyun Zhou, Jiao Shi, Yu Lei

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)

2605.08145 2026-06-01 cs.CV cs.AI cs.LG 82%

Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models

自描述多模态交互调优：放大可利用冗余以实现鲁棒的视觉语言模型

Yuriel Ryan, Hei Man Ip, Adriel Kuek, Paul Pu Liang, Roy Ka-Wei Lee

机构 * Singapore University of Technology and Design（新加坡科技设计大学）； DSO National Laboratories（国防部国家实验室）； Massachusetts Institute of Technology（麻省理工学院）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI

AI总结针对视觉语言模型中的幻觉和鲁棒性问题，提出自描述多模态交互调优方法，通过放大模态间冗余信息来补偿受损模态，并设计多模态交互门机制将独特交互转化为冗余交互，实验表明该方法可减少38.3%的视觉诱导错误并提升16.8%的一致性。

Comments Accepted to ICML 2026. Code: https://github.com/yurielryan/Multimodal-Interaction-Tuning

2512.13747 2025-12-17 cs.CV cs.AI 82%

Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making

为何文本占上风：视觉可能损害多模态医疗决策制定

Siyuan Dai, Lunxiao Li, Kun Zhao, Eardi Lila, Paul K. Crane, Heng Huang, Dongkuan Xu, Haoteng Tang, Liang Zhan

机构 * University of Texas Rio Grande Valley（德克萨斯大学里奥格兰德谷大学）； University of Pittsburgh（匹兹堡大学）； NC State University（北卡罗来纳州立大学）； University of Washington（华盛顿大学）； University of Maryland（马里兰大学）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI

AI总结本研究发现文本推理在医疗多模态决策中优于多模态输入，提出三种策略以提升多模态医疗决策能力。

Comments Accepted by ICDM 2025 the Workshop on Synergy of AI and Multimodal Biomedical Data Mining

2510.20291 2025-10-24 cs.CV cs.AI 82%

A Parameter-Efficient Mixture-of-Experts Framework for Cross-Modal Geo-Localization

LinFeng Li, Jian Zhao, Zepeng Yang, Yuhang Song, Bojun Lin, Tianle Zhang, Yuchen Yuan, Chi Zhang, Xuelong Li

机构 * The Institute of Artificial Intelligence (TeleAI), China Telecom（人工智能研究所（TeleAI），中国电信）； East China Normal University（东华师范大学）； National Tsing Hua University（国立清华大学）

专题命中图文多模态：cross-modal(title,abstract)；分类 cs.CV、cs.AI

Journal ref IROS 2025 Robosense Cross-Modal Drone Navigation Challenge first place

1. 图文多模态 4634 篇

Visual Language Model based Cross-modal Semantic Communication Systems

MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment

Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality

FLEUR: An Explainable Reference-Free Evaluation Metric for Image Captioning Using a Large Multimodal Model

MediFact at MEDIQA-M3G 2024: Medical Question Answering in Dermatology with Multimodal Learning

ViP-LLaVA: Making Large Multimodal Models Understand Arbitrary Visual Prompts

CatLIP: CLIP-level Visual Recognition Accuracy with 2.7x Faster Pre-training on Web-scale Image-Text Data

BattleAgent: Multi-modal Dynamic Emulation on Historical Battles to Complement Historical Analysis

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

Medical Image Synthesis via Fine-Grained Image-Text Alignment and Anatomy-Pathology Prompting

Polos: Multimodal Metric Learning from Human Feedback for Image Captioning

On the Performance of Multimodal Language Models

Apollo: Zero-shot MultiModal Reasoning with Multiple Experts

Differentiable Outlier Detection Enable Robust Deep Multimodal Analysis

CLIPTrans: Transferring Visual Knowledge with Pre-trained Models for Multimodal Machine Translation

M^4I: Multi-modal Models Membership Inference

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language

Vision-Language Pre-Training for Multimodal Aspect-Based Sentiment Analysis

Multimodal neural networks better explain multivoxel patterns in the hippocampus

A Corpus for English-Japanese Multimodal Neural Machine Translation with Comparable Sentences

X-LXMERT: Paint, Caption and Answer Questions with Multi-Modal Transformers

Learning Multilingual Word Embeddings Using Image-Text Data

Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

M3D-GAN: Multi-Modal Multi-Domain Translation with Universal Attention

ParNet: Position-aware Aggregated Relation Network for Image-Text matching

Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation

Semi-supervised Multimodal Hashing

Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Models

Why Text Prevails: Vision May Undermine Multimodal Medical Decision Making

A Parameter-Efficient Mixture-of-Experts Framework for Cross-Modal Geo-Localization