多模态大模型 - arXivDaily 专题

2205.14204 2022-10-24 cs.CV 83%

Multimodal Masked Autoencoders Learn Transferable Representations

Xinyang Geng, Hao Liu, Lisa Lee, Dale Schuurmans, Sergey Levine, Pieter Abbeel

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

2111.11430 2022-07-20 cs.CV 83%

Class-agnostic Object Detection with Multi-modal Transformer

Muhammad Maaz, Hanoona Rasheed, Salman Khan, Fahad Shahbaz Khan, Rao Muhammad Anwer, Ming-Hsuan Yang

专题命中图文多模态：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted at ECCV 2022

2201.06686 2022-06-07 cs.CV 83%

Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching

Hengcan Shi, Munawar Hayat, Jianfei Cai

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments 9 pages, 7 figures

2204.04014 2022-05-06 cs.CV cs.NE 83%

Multimodal Quasi-AutoRegression: Forecasting the visual popularity of new fashion products

Stefanos I. Papadopoulos, Christos Koutlis, Symeon Papadopoulos, Ioannis Kompatsiaris

专题命中图文多模态：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 14 pages, 4 figures

2204.05533 2022-04-28 cs.CL cs.SI 83%

How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image

Hyewon Choi, Yejun Yoon, Seunghyun Yoon, Kunwoo Park

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL

Comments 9 pages, 8 figures including appendix figure, 2 tables. Published in Findings of ACL workshop, CONSTRAINT 2022 (Long paper). The manuscript is slightly revised after the camera ready version

2204.04588 2022-04-12 cs.CV cs.LG 83%

Robust Cross-Modal Representation Learning with Progressive Self-Distillation

Alex Andonian, Shixing Chen, Raffay Hamid

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted to CVPR 2022

2202.06076 2022-02-15 cs.CV cs.LG 83%

Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers

Grzegorz Jacenków, Alison Q. O'Neil, Sotirios A. Tsaftaris

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted at the IEEE International Symposium on Biomedical Imaging (ISBI) 2022 as an oral presentation

2108.08217 2021-08-19 cs.CV cs.AI cs.CL cs.LG cs.MM 83%

X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics

Yehao Li, Yingwei Pan, Jingwen Chen, Ting Yao, Tao Mei

专题命中图文多模态：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by 2021 ACMMM Open Source Software Competition. Source code: https://github.com/YehLi/xmodaler

2107.14596 2021-08-02 cs.CL 83%

Multi-stage Pre-training over Simplified Multimodal Pre-training Models

Tongtong Liu, Fangxiang Feng, Xiaojie Wang

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL

Comments 10 pages, 4 figures

2107.11970 2021-07-27 cs.CV 83%

Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph

Wentian Zhao, Yao Hu, Heda Wang, Xinxiao Wu, Jiebo Luo

专题命中图文多模态：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2107.10300 2021-07-23 cs.CV cs.AI cs.CL cs.LG cs.MM eess.IV 83%

iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability

Aman Chadha, Vinija Jain

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 12 pages, 1 figure, 7 tables

2106.06509 2021-06-14 cs.CV 83%

Step-Wise Hierarchical Alignment Network for Image-Text Matching

Zhong Ji, Kexin Chen, Haoran Wang

专题命中图文多模态：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by IJCAI 2021

2104.12471 2021-04-27 cs.CV cs.AI cs.CL cs.IR cs.MM 83%

Contextualized Keyword Representations for Multi-modal Retinal Image Captioning

Jia-Hong Huang, Ting-Wei Wu, Marcel Worring

专题命中图文多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments This paper is accepted by ACM International Conference on Multimedia Retrieval (ICMR), 2021

2104.10406 2021-04-22 cs.CV 83%

Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching

Shiyang Yan, Li Yu, Yuan Xie

专题命中图文多模态：image-text(title,abstract)；multi-modal(abstract)；分类 cs.CV

2006.09310 2020-06-17 cs.CV cs.LG 83%

Deep Multimodal Transfer-Learned Regression in Data-Poor Domains

Levi McClenny, Mulugeta Haile, Vahid Attari, Brian Sadler, Ulisses Braga-Neto, Raymundo Arroyave

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

1908.06066 2019-12-04 cs.CV 83%

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Gen Li, Nan Duan, Yuejian Fang, Ming Gong, Daxin Jiang, Ming Zhou

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments accepted by AAAI-2020. arXiv admin note: text overlap with arXiv:1909.11740 by other authors

1907.12490 2019-07-30 cs.IR cs.MM 83%

Deep Cross-Modal Hashing with Hashing Functions and Unified Hash Codes Jointly Learning

Rong-Cheng Tu, Xian-Ling Mao, Bing Ma, Yong Hu, Tan Yan, Wei Wei, Heyan Huang

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.MM

1906.09610 2019-06-25 cs.CV 83%

Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments

Kai Niu, Yan Huang, Wanli Ouyang, Liang Wang

专题命中图文多模态：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV

1904.06286 2019-04-17 cs.CL 83%

CITE: A Corpus of Image-Text Discourse Relations

Malihe Alikhani, Sreyasi Nag Chowdhury, Gerard de Melo, Matthew Stone

专题命中图文多模态：image-text(title,abstract)；multimodal(abstract)；分类 cs.CL

Comments 7 pages

Journal ref 2019 Annual Conference of the North American Chapter of the Association for Computational Linguistics

1809.04144 2018-09-13 cs.CV 83%

End-to-end Image Captioning Exploits Multimodal Distributional Similarity

Pranava Madhyastha, Josiah Wang, Lucia Specia

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Published in BMVC 2018

1611.00471 2017-03-22 cs.CV 83%

Dual Attention Networks for Multimodal Reasoning and Matching

Hyeonseob Nam, Jung-Woo Ha, Jeonghee Kim

专题命中图文多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

1411.4738 2015-12-18 cs.MM cs.IR cs.LG 83%

Cross-Modal Similarity Learning : A Low Rank Bilinear Formulation

Cuicui Kang, Shengcai Liao, Yonghao He, Jian Wang, Wenjia Niu, Shiming Xiang, Chunhong Pan

专题命中图文多模态：cross-modal(title,abstract)；image-text(abstract)；分类 cs.MM

2504.21375 2025-05-01 cs.LG 83%

Synergy-CLIP: Extending CLIP with Multi-modal Integration for Robust Representation Learning

Sangyeon Cho, Jangyeong Jeon, Mingi Kim, Junyeong Kim

机构 * Department of Artificial Intelligence, Chung-Ang University, Seoul 06974, Republic of Korea（人工智能系，成均馆大学，韩国首尔）

专题命中图文多模态：multi-modal(title,abstract)；image-text(abstract)

Comments Multi-modal, Multi-modal Representation Learning, Missing Modality, Missing Modality Reconstruction, Speech and Multi-modality, Vision and Language

2510.22665 2026-08-11 cs.CV cs.AI 版本更新 82%

SARVLM: A Vision Language Foundation Model for Semantic Understanding in SAR Imagery

SARVLM：一种用于SAR图像语义理解的视觉语言基础模型

Qiwei Ma, Xukun Lu, Wang Liu, Puhong Duan, Xudong Kang, Shutao Li

机构 * School of Artificial Intelligence and Robotics, Hunan University（湖南大学人工智能与机器人学院）； Yuelushan Center for Industrial Innovation（岳麓山创新中心）； School of Medical Information Engineering, Jining Medical University（济南医学院医学信息工程学院）

专题命中图文多模态：multimodal(abstract)；multi-modal(abstract)；cross-modal(abstract)；image-text(abstract)

AI总结本文提出SARVLM，一种专为SAR图像设计的视觉语言基础模型，通过多阶段领域迁移策略和集成策略提升跨场景泛化能力，在多个基准测试中表现优异。

Comments 13 pages, 13 figures

2208.10442 2022-09-01 cs.CV cs.CL 82%

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, Furu Wei

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；multimodal foundation model(abstract)

Comments 18 pages

2608.11681 2026-08-13 cs.CV cs.AI cs.MM 新提交 82%

Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic Segmentation

学习多模态伪标签以实现鲁棒的开放词汇实例和全景分割

Duy Tran Thanh, Yeejin Lee, Byeongkeun Kang

机构 * Seoul National University of Science and Technology（首尔科技大学）； Chung-Ang University（中央大学）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

AI总结该研究针对开放词汇实例分割与开放集全景分割的痛点，提出多模态框架，结合预训练视觉语言模型生成伪标签并优化训练目标，在COCO数据集上取得优于现有SOTA的性能。

Comments 14 pages

Journal ref Neurocomputing, 2026

2605.16411 2026-08-12 cs.CV cs.AI cs.CL cs.DB cs.LG 版本更新 82%

Grounded Post-Training with Hard Examples for Reducing Hallucination in Multimodal Large Language Models

通过分布偏移下的分阶段偏好优化减少视觉-语言模型中的幻觉

Qinwu Xu

机构 * Meta AI

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结本文提出分阶段偏好优化框架，通过构建针对幻觉问题的数据集，提升视觉-语言模型的 grounded reasoning，减少幻觉并提高响应信息量。

2507.05515 2026-07-28 cs.AI cs.CL cs.CV 版本更新 82%

LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants

乐高协同构建器：探索用于多模态乐高组装助手的细粒度视觉语言建模

Haochen Huang, Yue Su, Xin Sun, Moonisa Ahsan, Mohammad Aliannejadi, Irene Viola, Zhaochun Ren, Chuang Yu, Aneta Lisowska, Artem Belopolsky, Koen Hindriks, Pablo Cesar, Junxiao Wang, Jiahuan Pei

机构 * Vrije University of Amsterdam University of Amsterdam Centrum Wiskunde \& Informatic ； University of Amsterdam National Institute of Informatics Centrum Wiskunde \& Informatic ； Centrum Wiskunde \& Informatic Leiden University University College London ； Vrije University of Amsterdam ； Centrum Wiskunde \& Informatica Technische Universiteit Delft Guangzhou University ； Vrije University of Amsterdam Centrum Wiskunde \& Informatic

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结研究针对视觉语言模型理解多模态组装指令的挑战，提出乐高协同构建器基准，引入统一框架评估多个VLMs及推理模型，发现物体检测性能高但细粒度场景理解和状态检测有挑战，还发布相关资源助力未来研究。

Comments This version has been accepted by ICMI 2026 Conference

2606.16484 2026-06-16 cs.CV cs.AI cs.MM 新提交 82%

Unified Multimodal Model for Brain MRI Imputation and Understanding

统一多模态模型用于脑MRI补全与理解

Zhiyun Song, Che Liu, Tian Xia, Avinash Kori, Wenjia Bai

机构 * Department of Computing, Imperial College London（伦敦帝国理工学院计算机系）； Department of Brain Sciences, Imperial College London（伦敦帝国理工学院脑科学系）

专题命中图文多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

AI总结提出UniBrain模型，通过统一训练策略联合处理脑MRI模态补全与图像理解，采用自对齐和动态隐藏状态机制，在多疾病数据集上实现高性能。

Comments Early accepted to MICCAI 2026

2606.09853 2026-06-10 cs.LG cs.IT math.IT 新提交 82%

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning

SynIB: 多模态学习中最大化协同的信息瓶颈

Konstantinos Kontras, Teodora Gagaleska, Thomas Strypsteen, Christos Chatzichristos, Matthew Blaschko, Maarten De Vos, Paul Pu Liang

机构 * University of California, Berkeley（加州大学伯克利分校）； Google Research（谷歌研究院）； University of Amsterdam（阿姆斯特丹大学）

专题命中图文多模态：multimodal(title,abstract)；cross-modal(abstract)

AI总结提出SynIB方法，通过信息瓶颈理论直接优化多模态协同，在训练中屏蔽单模态时惩罚高置信度，提升跨模态推理能力，在合成和真实任务上准确率提升达7.8%。

1. 图文多模态 4634 篇

Multimodal Masked Autoencoders Learn Transferable Representations

Class-agnostic Object Detection with Multi-modal Transformer

Unpaired Referring Expression Grounding via Bidirectional Cross-Modal Matching

Multimodal Quasi-AutoRegression: Forecasting the visual popularity of new fashion products

How does fake news use a thumbnail? CLIP-based Multimodal Detection on the Unrepresentative News Image

Robust Cross-Modal Representation Learning with Progressive Self-Distillation

Indication as Prior Knowledge for Multimodal Disease Classification in Chest Radiographs with Transformers

X-modaler: A Versatile and High-performance Codebase for Cross-modal Analytics

Multi-stage Pre-training over Simplified Multimodal Pre-training Models

Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph

iReason: Multimodal Commonsense Reasoning using Videos and Natural Language with Interpretability

Step-Wise Hierarchical Alignment Network for Image-Text Matching

Contextualized Keyword Representations for Multi-modal Retinal Image Captioning

Discrete-continuous Action Space Policy Gradient-based Attention for Image-Text Matching

Deep Multimodal Transfer-Learned Regression in Data-Poor Domains

Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-training

Deep Cross-Modal Hashing with Hashing Functions and Unified Hash Codes Jointly Learning

Improving Description-based Person Re-identification by Multi-granularity Image-text Alignments

CITE: A Corpus of Image-Text Discourse Relations

End-to-end Image Captioning Exploits Multimodal Distributional Similarity

Dual Attention Networks for Multimodal Reasoning and Matching

Cross-Modal Similarity Learning : A Low Rank Bilinear Formulation

Synergy-CLIP: Extending CLIP with Multi-modal Integration for Robust Representation Learning

SARVLM: A Vision Language Foundation Model for Semantic Understanding in SAR Imagery

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Learning from Multimodal Pseudo-Labels for Robust Open-Vocabulary Instance and Panoptic Segmentation

Grounded Post-Training with Hard Examples for Reducing Hallucination in Multimodal Large Language Models

LEGO Co-builder: Exploring Fine-Grained Vision-Language Modeling for Multimodal LEGO Assembly Assistants

Unified Multimodal Model for Brain MRI Imputation and Understanding

SynIB: Informational Bottleneck for Maximizing Synergy in Multimodal Learning