多模态大模型

CG-MLLM: Captioning and Generating 3D content via Multi-modal Large Language Models

CG-MLLM：通过多模态大语言模型实现图像描述与3D内容生成

Junming Huang, Chi Wang, Letian Li, Guangkai Xu, Donglin Huang, Hao Chen, Qiang Dai, Weiwei Xu

机构 * Zhejiang University, China（浙江大学） 纠错

专题命中 图文多模态 ：MLLM(title,title_cn)；multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

AI总结 本文提出CG-MLLM，一种能实现3D描述和高分辨率3D生成的多模态大语言模型，通过混合Transformer架构分离不同建模需求，结合预训练视觉语言模型与专用3D VAE潜在空间，提升3D生成质量与感知能力。

Comments ICML 2026

COHERENCE: Benchmarking Fine-Grained Image-Text Alignment in Interleaved Multimodal Contexts

COHERENCE：在交错多模态上下文中细粒度图像-文本对齐的基准测试

Bingli Wang, Huanze Tang, Haijun Lv, Zhishan Lin, Lixin Gu, Lei Feng, Qipeng Guo, Kai Chen

机构 * Southeast University Shanghai AI Laboratory（上海大学上海人工智能实验室） ； Shanghai AI Laboratory（上海人工智能实验室） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(title,abstract)；MLLM(summary_cn,abstract_cn)；分类 cs.CV、cs.AI

AI总结 COHERENCE旨在评估MLLM在交错多模态上下文中恢复细粒度图像-文本对应关系的能力，涵盖四个领域，包含6161个高质量问题，并进行六类错误分析以识别当前MLLM的不足。

Learning to Fuse: Modality-Aware Adaptive Scheduling for Robust Multimodal Foundation Models

Liam Bennett, Mason Clark, Lucas Anderson, Hana Satou, Olivia Martinez

机构 * Alan Mitkiy, Michael Johnson, Sofia García, Hana Satou（未知机构） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；cross-modal(abstract)；image-text(abstract)

MR-MLLM: Mutual Reinforcement of Multimodal Comprehension and Vision Perception

Guanqun Wang, Xinyu Wei, Jiaming Liu, Ray Zhang, Yichi Zhang, Kevin Zhang, Maurice Chong, Shanghang Zhang

专题命中 图文多模态 ：multimodal(title,abstract)；MLLM(title,abstract)；cross-modal(abstract)；image-text(abstract)

Comments 14 pages, 8 figures

Traj-MLLM: Can Multimodal Large Language Models Reform Trajectory Data Mining?

Shuo Liu, Di Yao, Yan Lin, Gao Cong, Jingping Bi

机构 * University of Chinese Academy of Sciences（中国科学院大学） ； Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所） ； Department of Computer Science, Aalborg University（奥胡斯大学计算机科学系） ； College of Computing and Data Science, Nanyang Technological University（南洋理工大学计算与数据科学学院） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；MLLM(title,abstract)；image-text(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments 20 pages, 10 figures

Multimodality Helps Unimodality: Cross-Modal Few-Shot Learning with Multimodal Models

Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, Deva Ramanan

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Published at CVPR 2023. Project site: https://linzhiqiu.github.io/papers/cross_modal/

Intern-S1-Pro: Scientific Multimodal Foundation Model at Trillion Scale

Intern-S1-Pro：万亿参数科学多模态基础模型

Yicheng Zou, Dongsheng Zhu, Lin Zhu, Tong Zhu, Yunhua Zhou, Peiheng Zhou, Xinyu Zhou, Dongzhan Zhou, Zhiwang Zhou, Yuhao Zhou, Bowen Zhou, Zhanping Zhong, Zhijie Zhong, Haiteng Zhao, Penghao Zhao, Xiaomeng Zhao, Zhiyuan Zhao, Yechen Zhang, Jin Zhang, Wenwei Zhang, Hongjie Zhang, Zhuo Zhang, Wenlong Zhang, Bo Zhang, Chao Zhang, Chen Zhang, Yuhang Zang, Fei Yuan, Jiakang Yuan, Jiashuo Yu, Jinhui Yin, Haochen Ye, Qian Yao, Bowen Yang, Danni Yang, Kaichen Yang, Ziang Yan, Jun Xu, Yicheng Xu, Wanghan Xu, Xuenan Xu, Chao Xu, Ruiliang Xu, Shuhao Xing, Long Xing, Xinchen Xie, Ling-I Wu, Zijian Wu, Zhenyu Wu, Lijun Wu, Yue Wu, Jianyu Wu, Wen Wu, Fan Wu, Xilin Wei, Qi Wei, Bingli Wang, Rui Wang, Ziyi Wang, Zun Wang, Yi Wang, Haomin Wang, Yizhou Wang, Lintao Wang, Yiheng Wang, Longjiang Wang, Bin Wang, Jian Tong, Zhongbo Tian, Huanze Tang, Chen Tang, Shixiang Tang, Yu Sun, Qiushi Sun, Xuerui Su, Qisheng Su, Chenlin Su, Demin Song, Jin Shi, Fukai Shang, Yuchen Ren, Pengli Ren, Xiaoye Qu, Yuan Qu, Jiantao Qiu, Yu Qiao, Biqing Qi, Runyu Peng, Tianshuo Peng, Jiahui Peng, Qizhi Pei, Zhuoshi Pan, Linke Ouyang, Wenchang Ning, Yichuan Ma, Zerun Ma, Ningsheng Ma, Runyuan Ma, Chengqi Lyu, Haijun Lv, Han Lv, Lindong Lu, Kuikun Liu, Jiangning Liu, Yuhong Liu, Kai Liu, Hongwei Liu, Zhoumianze Liu, Mengjie Liu, Ziyu Liu, Wenran Liu, Yang Liu, Liwei Liu, Kaiwen Liu, Junyao Lin, Junming Lin, Tianyang Lin, Dahua Lin, Jianze Liang, Linyang Li, Peiji Li, Zonglin Li, Zehao Li, Pengze Li, Guoyan Li, Lingkai Kong, Linglin Jing, Zhenjiang Jin, Feifei Jiang, Qian Jiang, Junhao Huang, Zixian Huang, Haian Huang, Zhouqi Hua, Ermo Hua, Han Hu, Linfeng Hou, Yinan He, Conghui He, Tianyao He, Xu Guo, Qipeng Guo, Aijia Guo, Yuzhe Gu, Lixin Gu, Jingyang Gong, Qiming Ge, Jiaye Ge, Songyang Gao, Jianfei Gao, Xinyu Fang, Caihua fan, Yue Fan, Yanhui Duan, Zichen Ding, Shengyuan Ding, Ning Ding, Xuanlang Dai, Erfei Cui, Ganqu Cui, Pei Chu, Tao Chu, Guangran Cheng, Yu Cheng, Kai Chen, Yongkang Chen, Chiyu Chen, Guanzhou Chen, Qiaosheng Chen, Sitao Chen, Xin Chen, Haojiong Chen, Yicheng Chen, Weihan Cao, Yuhang Cao, Qinglong Cao, Lei Bai

机构 * Shanghai AI Laboratory（上海人工智能实验室） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

AI总结 Intern-S1-Pro是首个万亿参数科学多模态基础模型，具备跨通用与科学领域的能力提升，融合强推理、图像-文本理解及先进代理能力，专精于100余项科学任务。

Evaluating Cross-Modal Reasoning Ability and Problem Characteristics with Multimodal Item Response Theory

利用多模态项目反应理论评估跨模态推理能力与问题特征

Shunki Uebayashi, Kento Masui, Kyohei Atarashi, Han Bao, Hisashi Kashima, Naoto Inoue, Mayu Otani, Koh Takeuchi

机构 * Kyoto University（京都大学） ； CyberAgent ； The Institute of Statistical Mathematics（统计数学研究所） ； Tohoku University（东北大学） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

AI总结 M3IRT通过分解模型能力和项目难度，提供了一种评估多模态推理能力的框架，有效提升基准测试的可靠性和质量。

Comments 24pages, 20 figures, accepted to ICLR2026

Distill CLIP (DCLIP): Enhancing Image-Text Retrieval via Cross-Modal Transformer Distillation

Daniel Csizmadia, Andrei Codreanu, Victor Sim, Vighnesh Prabhu, Michael Lu, Kevin Zhu, Sean O'Brien, Vasu Sharma

机构 * Algoverse AI Research（Algoverse AI研究院） ； Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 图文多模态 ：cross-modal(title,abstract)；image-text(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs

Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, Cliff Wong, Andrea Tupini, Yu Wang, Matt Mazzola, Swadheen Shukla, Lars Liden, Jianfeng Gao, Angela Crabtree, Brian Piening, Carlo Bifulco, Matthew P. Lungren, Tristan Naumann, Sheng Wang, Hoifung Poon

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(title,abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.CL

Comments The models are released at https://aka.ms/biomedclip

Revisit Large-Scale Image-Caption Data in Pre-training Multimodal Foundation Models

Zhengfeng Lai, Vasileios Saveris, Chen Chen, Hong-You Chen, Haotian Zhang, Bowen Zhang, Juan Lao Tebar, Wenze Hu, Zhe Gan, Peter Grasch, Meng Cao, Yinfei Yang

专题命中 图文多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments CV/ML

Advanced Multimodal Deep Learning Architecture for Image-Text Matching

Jinyin Wang, Haijing Zhang, Yihao Zhong, Yingbin Liang, Rongwei Ji, Yiru Cang

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments arXiv admin note: text overlap with arXiv:2405.17460 by other authors

Multimodal foundation models are better simulators of the human brain

Haoyu Lu, Qiongyi Zhou, Nanyi Fei, Zhiwu Lu, Mingyu Ding, Jingyuan Wen, Changde Du, Xin Zhao, Hao Sun, Huiguang He, Ji-Rong Wen

专题命中 图文多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；image-text(abstract)；分类 cs.AI、cs.MM

VLT: A Vision-Language-Time Series Multimodal Foundation Model for Industrial Intelligence

VLT：用于工业智能的视觉-语言-时间序列多模态基础模型

Haiteng Wang, Jingheng Yan, Xiaokang Wang, Lei Ren

机构 * School of Automation Science and Electrical Engineering, Beihang University（北京航空航天大学自动化科学与电气工程学院） ； Hangzhou International Innovation Institute, Beihang University（北京航空航天大学杭州国际创新研究院） ； State Key Laboratory of Intelligent Manufacturing System Technology（智能制造系统技术国家重点实验室） ； School of Computer Science and Artificial Intelligence, Zhengzhou University（郑州大学计算机科学与人工智能学院） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；cross-modal(abstract)；分类 cs.AI

AI总结 针对工业时间序列单模态建模局限及连接时间序列与文本语义的挑战，提出VLT多模态基础模型，通过设计Time-MoE等机制联合建模多种模态，经实验验证其在多种复杂设置下优于现有方法，提升了鲁棒性和泛化能力。

Comments 18 pages, 13 figures, and 13 tables, including supplementary material. Haiteng Wang and Jingheng Yan contributed equally to this work

Brain-Inspired Multimodal Spiking Neural Network for Image-Text Retrieval

脑启发式多模态脉冲神经网络用于图像-文本检索

Xintao Zong, Xian Zhong, Wenxuan Liu, Jianhao Ding, Zhaofei Yu, Tiejun Huang

机构 * Hubei Key Laboratory of Transportation Internet of Things, Wuhan University of Technology（武汉理工大学交通物联网湖北省重点实验室） ； State Key Laboratory for Multimedia Information Processing, Peking University（北京大学多媒体信息处理全国重点实验室） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV

AI总结 本文提出脑启发式跨模态脉冲融合网络(CMSF)，用于图像-文本检索，通过融合单模态特征提升多模态表示，实现高效检索与低能耗。

Red-teaming the Multimodal Reasoning: Jailbreaking Vision-Language Models via Cross-modal Entanglement Attacks

多模态推理的红队测试：通过跨模态纠缠攻击劫持视觉-语言模型

Yu Yan, Sheng Sun, Shengjia Cheng, Teli Liu, Mingfeng Li, Min Liu

机构 * Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所） ； University of Chinese Academy of Sciences（中国科学院大学） ； People’s Public Security University of China（中国人民公安大学） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(title,abstract)；image-text(abstract)；分类 cs.AI

AI总结 本文提出CrossTALK方法，通过跨模态纠缠攻击提升对视觉-语言模型的劫持效果，实现高攻击成功率。

MUCAR: Benchmarking Multilingual Cross-Modal Ambiguity Resolution for Multimodal Large Language Models

Xiaolong Wang, Zhaolu Kang, Wangyuxuan Zhai, Xinyue Lou, Yunghwei Lai, Ziyue Wang, Yawen Wang, Kaiyu Huang, Yile Wang, Peng Li, Yang Liu

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(title,abstract)；image-text(abstract)；分类 cs.CL

Enhancing Remote Sensing Vision-Language Models Through MLLM and LLM-Based High-Quality Image-Text Dataset Generation

Yiguo He, Junjie Zhu, Yiying Li, Xiaoyu Zhang, Chunping Qiu, Jun Wang, Qiangjuan Huang, Ke Yang

机构 * Intelligent Game and Decision Lab (IGDL)（智能游戏与决策实验室） 纠错

专题命中 图文多模态 ：MLLM(title,abstract)；image-text(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments SUBMIT TO IEEE TRANSACTIONS

OCC-MLLM:Empowering Multimodal Large Language Model For the Understanding of Occluded Objects

Wenmo Qiu, Xinhan Di

专题命中 图文多模态 ：multimodal(title,abstract)；MLLM(title)；multi-modal(abstract)；image-text(abstract)

Comments Accepted by CVPR 2024 T4V Workshop (5 pages, 3 figures, 2 tables)

Improving Multimodal Classification of Social Media Posts by Leveraging Image-Text Auxiliary Tasks

Danae Sánchez Villegas, Daniel Preoţiuc-Pietro, Nikolaos Aletras

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments Accepted at EACL 2024 Findings

An Unsupervised Cross-Modal Hashing Method Robust to Noisy Training Image-Text Correspondences in Remote Sensing

Georgii Mikriukov, Mahdyar Ravanbakhsh, Begüm Demir

专题命中 图文多模态 ：cross-modal(title,abstract)；image-text(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments https://git.tu-berlin.de/rsim/chnr

Cross-modal Scene Graph Matching for Relationship-aware Image-Text Retrieval

Sijin Wang, Ruiping Wang, Ziwei Yao, Shiguang Shan, Xilin Chen

专题命中 图文多模态 ：cross-modal(title,abstract)；image-text(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by WACV 2020

Cascaded Sparse Autoencoders Learn Multi-Level Visual Concepts in Multimodal LLMs

级联稀疏自编码器在多模态大语言模型中学习多级视觉概念

Yusong Zhao, Hengyi Wang, Tanuja Ganu, Akshay Nambi, Hao Wang

机构 * Rutgers University（罗格斯大学） ； Microsoft Research（微软研究院） 纠错

专题命中 图文多模态 ：MLLM(summary_cn,abstract)；multimodal(title,abstract)；分类 cs.CV、cs.AI

AI总结 提出级联稀疏自编码器（CSAEs），通过在第一级SAE解码器权重上训练第二级SAE来学习层次化视觉概念，避免嵌套或堆叠SAE的缺点，在多个MLLM和数据集上提升了概念层次一致性和干预效果。

Multimodal Distribution Matching for Vision-Language Dataset Distillation

多模态分布匹配用于视觉-语言数据集蒸馏

Jongoh Jeong, Hoyong Kwon, Minseok Kim, Kuk-Jin Yoon

机构 * Visual Intelligence Lab., KAIST（韩国科学技术院视觉智能实验室） 纠错

专题命中 图文多模态 ：multimodal(title,summary_cn)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 提出多模态分布匹配（MDM）框架，通过数据、模型和损失层面的几何感知组件，高效压缩视觉-语言数据集并保持跨模态对齐。

Comments Accepted for publication at CVPR 2026. Project Page: https://andyj1.github.io/mdm

A Survey on Image-text Multimodal Models

Ruifeng Guo, Jingxuan Wei, Linzhuang Sun, Bihui Yu, Guiyong Chang, Dawei Liu, Sibo Zhang, Zhengbing Yao, Mingjun Xu, Liping Bu

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Hate-CLIPper: Multimodal Hateful Meme Classification based on Cross-modal Interaction of CLIP Features

Gokul Karthik Kumar, Karthik Nandakumar

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted at EMNLP 2022 Workshop on NLP for Positive Impact

Revising Image-Text Retrieval via Multi-Modal Entailment

Xu Yan, Chunhui Ai, Ziqiang Cao, Min Cao, Sujian Li, Wenjie Li, Guohong Fu

专题命中 图文多模态 ：multi-modal(title,abstract)；image-text(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 10 pages

Failure-Informed Image Self-Augmentation for Multimodal Large Language Model Self-Improvement

面向多模态大语言模型自我改进的故障感知图像自增强

Chunyang Jiang, Pingping Zhang, Yuzhi Zhao, Wenao Ma, Zhijian Hou, Mengyang Wu, Yiyang Cai, Senkang Hu, Sitong Cheng, Chi-Min Chan, Wei Xue, Yike Guo

专题命中 图文多模态 ：MLLM(summary_cn,abstract)；multimodal(title,abstract)；分类 cs.AI

AI总结 提出FISA框架，基于MLLM失败案例生成保留答案的增强图像，经自检验与双重保真过滤后，可提升视觉问答性能，且兼容文本自增强、数据效率更优。

From Geometric Labels to Semantic Understanding of Indoor Building Components Using Multimodal Large Language Models

使用多模态大语言模型从几何标签到室内建筑组件的语义理解

Shuju Jing, Chao Yin

机构 * School of Qilu Transportation, Shandong University（山东大学齐鲁交通学院） ； Guangzhou Institute of Geography, Guangdong Academy of Sciences（广东省地理研究所） 纠错

专题命中 图文多模态 ：MLLM(summary_cn,abstract)；multimodal(title,abstract)；分类 cs.CV

AI总结 研究针对室内建筑组件，提出以点云为中心的多模态大语言模型Building-MLLM，通过特定机制解决语义集中问题，开发引擎编译数据集，实验表明其在多任务上表现优异，提升室内组件语言理解。

Comments 46 pages, 13 figures, accepted by Automation in Construction journal

From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Models

从结构到协同：多模态大语言模型中视觉-语言感知范式演进综述

Haoxiang Sun, Tao Wang, Li Yuan, Jian Zhao, Jiancheng Lv

机构 * School of Computer Science, Sichuan University（四川大学计算机学院） ； School of Electronic and Computer Engineering, Peking University Shenzhen Graduate School（北京大学深圳研究生院电子与计算机工程学院） ； Institute of Artificial Intelligence (TeleAI), China Telecom and Northwestern Polytechnical University（中国电信与西北工业大学人工智能研究院（TeleAI）） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；MLLM(abstract,abstract_cn)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结 本文系统综述多模态大语言模型中统一视觉-语言感知的范式演进，提出五阶段分类法，梳理各阶段代表性方法，并指出开放挑战与未来方向。