多模态大模型

Unaligning Everything: Or Aligning Any Text to Any Image in Multimodal Models

Shaeke Salman, Md Montasir Bin Shams, Xiuwen Liu

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 14 pages, 14 figures. arXiv admin note: substantial text overlap with arXiv:2401.15568, arXiv:2402.08473

VEGA: Learning Interleaved Image-Text Comprehension in Vision-Language Large Models

Chenyu Zhou, Mengdan Zhang, Peixian Chen, Chaoyou Fu, Yunhang Shen, Xiawu Zheng, Xing Sun, Rongrong Ji

专题命中 图文多模态 ：image-text(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://zhourax.github.io/VEGA/

Groma: Localized Visual Tokenization for Grounding Multimodal Large Language Models

Chuofan Ma, Yi Jiang, Jiannan Wu, Zehuan Yuan, Xiaojuan Qi

专题命中 图文多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

CapsFusion: Rethinking Image-Text Data at Scale

Qiying Yu, Quan Sun, Xiaosong Zhang, Yufeng Cui, Fan Zhang, Yue Cao, Xinlong Wang, Jingjing Liu

专题命中 图文多模态 ：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2024. Code & Dataset: https://github.com/baaivision/CapsFusion

Towards Unified Multi-Modal Personalization: Large Vision-Language Models for Generative Recommendation and Beyond

Tianxin Wei, Bowen Jin, Ruirui Li, Hansi Zeng, Zhengyang Wang, Jianhui Sun, Qingyu Yin, Hanqing Lu, Suhang Wang, Jingrui He, Xianfeng Tang

专题命中 图文多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments ICLR 2024

HyperLLaVA: Dynamic Visual and Language Expert Tuning for Multimodal Large Language Models

Wenqiao Zhang, Tianwei Lin, Jiang Liu, Fangxun Shu, Haoyuan Li, Lei Zhang, He Wanggui, Hao Zhou, Zheqi Lv, Hao Jiang, Juncheng Li, Siliang Tang, Yueting Zhuang

专题命中 图文多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Toward Robust Multimodal Learning using Multimodal Foundational Models

Xianbing Zhao, Soujanya Poria, Xuejiao Li, Yixin Chen, Buzhou Tang

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Under Review

Structure-CLIP: Towards Scene Graph Knowledge to Enhance Multi-modal Structured Representations

Yufeng Huang, Jiji Tang, Zhuo Chen, Rongsheng Zhang, Xinfeng Zhang, Weijie Chen, Zeng Zhao, Zhou Zhao, Tangjie Lv, Zhipeng Hu, Wen Zhang

专题命中 图文多模态 ：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments AAAI 2024, https://github.com/zjukg/Structure-CLIP

Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception

Hassan Akbari, Dan Kondratyuk, Yin Cui, Rachel Hornung, Huisheng Wang, Hartwig Adam

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI、cs.MM

A New Fine-grained Alignment Method for Image-text Matching

Yang Zhang

专题命中 图文多模态 ：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models

Ziyi Lin, Chris Liu, Renrui Zhang, Peng Gao, Longtian Qiu, Han Xiao, Han Qiu, Chen Lin, Wenqi Shao, Keqin Chen, Jiaming Han, Siyuan Huang, Yichi Zhang, Xuming He, Hongsheng Li, Yu Qiao

专题命中 图文多模态 ：multi-modal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Work in progress. Code and demos are released at https://github.com/Alpha-VLLM/LLaMA2-Accessory

MAP: Multimodal Uncertainty-Aware Vision-Language Pre-training Model

Yatai Ji, Junjie Wang, Yuan Gong, Lin Zhang, Yanru Zhu, Hongfa Wang, Jiaxing Zhang, Tetsuya Sakai, Yujiu Yang

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments CVPR 2023 Main Track Long Paper

MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learning

Zijia Zhao, Longteng Guo, Xingjian He, Shuai Shao, Zehuan Yuan, Jing Liu

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments SIGIR 2023, 10 pages

Is Multimodal Vision Supervision Beneficial to Language?

Avinash Madasu, Vasudev Lal

专题命中 图文多模态 ：multimodal(title)；multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

MoMo: A shared encoder Model for text, image and multi-Modal representations

Rakesh Chada, Zhaoheng Zheng, Pradeep Natarajan

专题命中 图文多模态 ：multi-modal(title)；multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillation

Wenliang Dai, Lu Hou, Lifeng Shang, Xin Jiang, Qun Liu, Pascale Fung

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to ACL 2022

LoopITR: Combining Dual and Cross Encoder Architectures for Image-Text Retrieval

Jie Lei, Xinlei Chen, Ning Zhang, Mengjiao Wang, Mohit Bansal, Tamara L. Berg, Licheng Yu

专题命中 图文多模态 ：image-text(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships

通过利用一对一关系进行多模态对抗防御以提升视觉语言模型

Futa Waseda, Antonio Tejero-de-Pablos, Isao Echizen

机构 * The University of Tokyo（东京大学） ； CyberAgent ； National Institute of Informatics（信息处理研究所） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 本文提出多模态对抗训练方法，通过利用图像与文本之间的一对多关系提升视觉语言模型的对抗鲁棒性。

Comments WACV 2026 Accepted. Code available at https://github.com/CyberAgentAILab/multimodal-adversarial-training

Advances in Multimodal Adaptation and Generalization: From Traditional Approaches to Foundation Models

Hao Dong, Moru Liu, Kaiyang Zhou, Eleni Chatzi, Juho Kannala, Cyrill Stachniss, Olga Fink

专题命中 图文多模态 ：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.AI

Comments Project page: https://github.com/donghao51/Awesome-Multimodal-Adaptation

On Domain-Adaptive Post-Training for Multimodal Large Language Models

Daixuan Cheng, Shaohan Huang, Ziyu Zhu, Xintong Zhang, Wayne Xin Zhao, Zhongzhi Luan, Bo Dai, Zhenliang Zhang

机构 * BIGAI ； BUAA（北京航空航天大学） ； THU（清华大学） ； BIT（北京理工大学） ； RUC（中国人民大学） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；MLLM(abstract,comments)；分类 cs.CV、cs.CL

Comments EMNLP 2025 Findings, Project Page: https://huggingface.co/AdaptLLM/Adapt-MLLM-to-Domains

DGSeg: Dynamic Gating of Semantic-Spatial Guided Predictions for Reasoning Segmentation

DGSeg：用于推理分割的语义-空间引导预测的动态门控

Ruizhe Zeng, Siyu Cao, Lu Zhang, Zhiyong Liu

机构 * State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院） ； Nanjing Artificial Intelligence Research of IA（中国科学院自动化所南京人工智能研究院） 纠错

专题命中 图文多模态 ：MLLM(summary_cn,abstract)；multimodal(abstract)；分类 cs.CV

AI总结 研究推理分割问题，提出DGSeg框架，借助MLLM生成语义和空间线索，经动态门控模块自适应融合预测，抑制噪声或冲突区域，在多基准测试中表现出色。

Comments Accepted to ECCV2026

Estimating Uncertainty in Multimodal Foundation Models using Public Internet Data

Shiladitya Dutta, Hongbo Wei, Lars van der Laan, Ahmed M. Alaa

专题命中 图文多模态 ：multimodal(title)；multimodal foundation model(title)；分类 cs.AI

The Role of Natural Language Understanding in Multimodal Video-Based Dengue Diagnosis

自然语言理解在基于多模态视频的登革热诊断中的作用

Danial Sharifrazi, Saadat Behzadi, Julakha Jahan Jui, Mojtaba Mohammadi, Nouman Javed, Roohallah Alizadehsani, Prasad N. Paradkar, Asim Bhatti

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 本研究提出基于YOLO和CLIP的视觉-语言框架，从视频中分类未受感染与DENV2感染的蚊子，帧级准确率达98.54%、灵敏度达99.91%，证实该框架可用于分析感染相关生物行为。

RobustVLA: On Robustness of Vision-Language-Action Model against Multi-Modal Perturbations

针对多模态扰动的视觉-语言-动作模型的鲁棒性

Jianing Guo, Zhenhong Wu, Chang Tu, Yiyao Ma, Xiangqi Kong, Zhiqian Liu, Jiaming Ji, Shuning Zhang, Yuanpei Chen, Kai Chen, Qi Dou, Yaodong Yang, Xianglong Liu, Huijie Zhao, Weifeng Lv, Simin Li

专题命中 图文多模态 ：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

AI总结 本研究提出RobustVLA，通过多模态鲁棒优化提升视觉-语言-动作模型在多种扰动下的鲁棒性，实验表明其在性能和推理速度上均优于现有方法。

Comments Published as a conference paper at ICLR 2026

SpectralGCD: Spectral Concept Selection and Cross-modal Representation Learning for Generalized Category Discovery

SpectralGCD：用于广义类别发现的光谱概念选择和跨模态表示学习

Lorenzo Caselli, Marco Mistretta, Simone Magistri, Andrew D. Bagdanov

机构 * University of Florence, Media Integration and Communication Center (MICC)（佛罗伦萨大学，媒体整合与通信中心（MICC）） 纠错

专题命中 图文多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

AI总结 研究广义类别发现问题，提出SpectralGCD方法，利用CLIP跨模态相似性统一表示，通过光谱滤波和知识蒸馏提升性能，相比现有方法计算成本低且精度高。

Comments Accepted at ICLR 2026. Code available at https://github.com/miccunifi/SpectralGCD

SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoning

SyncLoop: 一种用于自我改进数学推理的多模态双循环框架

Xiuwei Chen, Wentao Hu, Hanhui Li, Yongxin Wang Jun Zhou, Zisheng Chen, Meng Cao, Yihan Zeng, Kui Zhang, Yu-Jie Yuan, Jianhua Han, Hang Xu, Xiaodan Liang

机构 * Sun Yat-sen University（中山大学） ； The Hong Kong Polytechnic University（香港理工大学） ； MBZUAI ； Huawei Noah’s Ark Lab（华为诺亚实验室） ； Yinwang Intelligent Technology Co. Ltd（依文智科有限公司） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

AI总结 提出SyncLoop多模态双循环框架，通过跨模态数据演化循环和数据-模型演化循环联合进化训练数据和模型能力，解决数据复杂度不匹配和演化分离问题，在数学推理基准上持续提升性能。

Comments ECCV2026

CFPO: Counterfactual Policy Optimization for Multimodal Reasoning

CFPO：用于多模态推理的反事实策略优化

Zhangyuan Yu, Wanran Sun, Guangjing Yang, Xiaohu Wu, Qicheng Lao

机构 * Beijing University of Posts and Telecommunications（北京邮电大学） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

AI总结 提出反事实策略优化框架，通过跨模态反事实增强机制强制视觉与文本推理的因果一致性，显著提升多模态推理的准确性。

Comments Accepted to ICML 2026. 17 pages

Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?

受神经启发的多模态视觉-语言模型对成员推断隐私泄露是否具有弹性？

David Amebley, Sayanton Dibbo

机构 * The University of Alabama（阿拉巴马大学） ； Alabama Center for the Advancement of AI（阿拉巴马人工智能 advancement 中心） ； Trustworthy AI Lab（可信人工智能实验室） ； Department of Computer Science, The University of Alabama（计算机科学系） 纠错

专题命中 图文多模态 ：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 研究受神经启发的多模态视觉-语言模型（VLM）对基于图像-文本的成员推断攻击的弹性，提出拓扑正则化框架，实验表明神经VLM在保持模型效用同时显著降低攻击成功率。

Comments Accepted at USENIX WOOT '26

Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning

Bad Seeing or Bad Thinking? Rewarding Perception for Multimodal Reasoning

Haozhe Wang, Qixin Xu, Changpeng Wang, Taofeng Xue, Chong Peng, Wenhu Chen, Fangzhen Lin

机构 * University of Science and Technology of China（中国科学技术大学） 纠错

专题命中 图文多模态 ：multimodal(title,title_cn)；分类 cs.CV、cs.AI

AI总结 提出一种基于强化学习的模态感知信用分配框架（MoCA），通过感知验证和结构化口头验证解决视觉语言模型中感知与推理的权衡问题，实现多任务性能提升。

Comments Accepted by ICML 2026 as Oral

DenseMLLM: Standard Multimodal LLMs for Dense Prediction

DenseMLLM：用于密集预测的标准多模态大语言模型

Yi Li, Hongze Shen, Lexiang Tang, Xin Li, Xinpeng Ding, Yinsong Liu, Deqiang Jiang, Xing Sun, Xiaomeng Li

机构 * Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, China（香港科技大学电子与计算机工程系） ； Tencent, Youtu-Lab, China（腾讯优图实验室） 纠错

专题命中 图文多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

AI总结 提出DenseMLLM，通过标准多模态大语言模型架构和视觉令牌监督策略，无需任务特定解码器即可实现语义分割、深度估计等密集预测任务，在多个基准上取得竞争性能。

Comments ICML 2026