多模态大模型

QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation

Ahmed Wasfy, Omer Nacar, Abdelakreem Elkhateb, Mahmoud Reda, Omar Elshehy, Adel Ammar, Wadii Boulila

机构 * NAMAA ； KAND CA Corp. ； Prince Sultan University 纠错

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding

Yeongjae Cho, Keonwoo Kim, Taebaek Hwang, Sungzoon Cho

机构 * Seoul National University（首尔国立大学） ； Kim & Chang AI&IT System Center（金·昌AI&IT系统中心） ； Waddle Corporation（Waddle公司） 纠错

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

A Multimodal Pipeline for Clinical Data Extraction: Applying Vision-Language Models to Scans of Transfusion Reaction Reports

Henning Schäfer, Cynthia S. Schmidt, Johannes Wutzkowsky, Kamil Lorek, Lea Reinartz, Johannes Rückert, Christian Temme, Britta Böckmann, Peter A. Horn, Christoph M. Friedrich

机构 * Institute for Transfusion Medicine, University Hospital Essen, Hufelandstraße 55, Essen, Germany（1 输血医学研究所，埃森大学医院，Hufelandstraße 55，德国） ； Department of Computer Science, University of Applied Sciences and Arts Dortmund (FHDO), Emil-Figge Str. 42, Dortmund, Germany（2 计算机科学系，多特蒙德应用科学大学（FHDO），Emil-Figge Str. 42，德国） ； Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), University Hospital Essen, Hufelandstraße 55, Essen, Germany（3 医学信息学、生物统计学和流行病学研究所（IMIBE），埃森大学医院，Hufelandstraße 55，德国） ； Institute for AI in Medicine (IKIM), University Hospital Essen, Girardetstraße 2, Essen, Germany（4 医学人工智能研究所（IKIM），埃森大学医院，Girardetstraße 2，德国） ； Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Hufelandstraße 55, Essen, Germany（5 干预性和诊断放射学及神经放射学研究所，埃森大学医院，Hufelandstraße 55，德国） ； Department of Dermatology, University Hospital Essen, Hufelandstraße 55, Essen, Germany（6 皮肤科系，埃森大学医院，Hufelandstraße 55，德国） 纠错

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.CL

GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing

Zilun Zhang, Haozhan Shen, Tiancheng Zhao, Bin Chen, Zian Guan, Yuhao Wang, Xu Jia, Yuxiang Cai, Yongheng Shang, Jianwei Yin

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

Multimodal Cognitive Reframing Therapy via Multi-hop Psychotherapeutic Reasoning

Subin Kim, Hoonrae Kim, Heejin Do, Gary Geunbae Lee

专题命中 图文多模态 ：multimodal(title)；分类 cs.CL、cs.AI

Comments NAACL 2025 Main

A Multi-Modal Explainability Approach for Human-Aware Robots in Multi-Party Conversation

Iveta Bečková, Štefan Pócoš, Giulia Belgiovine, Marco Matarese, Omar Eldardeer, Alessandra Sciutti, Carlo Mazzola

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CL、cs.AI

Comments 32pp (+6pp sup.mat.) Accepted in Computer Vision and Image Understanding Journal on January 23, 2025. This research received funding Horizon-Europe TERAIS project (G.A. 101079338) and Slovak Research and Development Agency, project no. APVV-21-0105

MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation

Siddharth Joshi, Besmira Nushi, Vidhisha Balachandran, Varun Chandrasekaran, Vibhav Vineet, Neel Joshi, Baharan Mirzasoleiman

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.CL

Clustering-based Image-Text Graph Matching for Domain Generalization

Nokyung Park, Daewon Chae, Jeongyong Shim, Sangpil Kim, Eun-Sol Kim, Jinkyu Kim

专题命中 图文多模态 ：image-text(title)；分类 cs.CV、cs.AI

RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Models

Seulki Park, Daeho Um, Hajung Yoon, Sanghyuk Chun, Sangdoo Yun, Jin Young Choi

专题命中 图文多模态 ：image-text(title)；分类 cs.CV、cs.AI

Comments Accepted to ECCV Synthetic Data for Computer Vision Workshop (Oral)

Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines

Zhixin Zhang, Yiyuan Zhang, Xiaohan Ding, Xiangyu Yue

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

Comments Code is available at https://github.com/cnzzx/VSA

HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adapter

Yumiao Zhao, Bo Jiang, Xiao Wang, Qin Xu, Jin Tang

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.MM

FairPIVARA: Reducing and Assessing Biases in CLIP-Based Multimodal Models

Diego A. B. Moreira, Alef Iury Ferreira, Jhessica Silva, Gabriel Oliveira dos Santos, Luiz Pereira, João Medrado Gondim, Gustavo Bonil, Helena Maia, Nádia da Silva, Simone Tiemi Hashiguti, Jefersson A. dos Santos, Helio Pedrini, Sandra Avila

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

Comments 14 pages, 10 figures. Accepted to 35th British Machine Vision Conference (BMVC 2024), Workshop on Privacy, Fairness, Accountability and Transparency in Computer Vision

BCAmirs at SemEval-2024 Task 4: Beyond Words: A Multimodal and Multilingual Exploration of Persuasion in Memes

Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Lele Wang, Giuseppe Carenini

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.CL

Comments 12 pages, 5 tables, 2 figures, Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024) @ NAACL 2024

A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction

Yu Hao, Fan Yang, Hao Huang, Shuaihang Yuan, Sundeep Rangan, John-Ross Rizzo, Yao Wang, Yi Fang

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.AI

Let's Go Shopping (LGS) -- Web-Scale Image-Text Dataset for Visual Concept Understanding

Yatong Bai, Utsav Garg, Apaar Shanker, Haoming Zhang, Samyak Parajuli, Erhan Bas, Isidora Filipovic, Amelia N. Chu, Eugenia D Fomitcheva, Elliot Branson, Aerin Kim, Somayeh Sojoudi, Kyunghyun Cho

专题命中 图文多模态 ：image-text(title)；分类 cs.CV、cs.AI

Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using Self-Imagination

Syeda Nahida Akter, Aman Madaan, Sangwu Lee, Yiming Yang, Eric Nyberg

专题命中 图文多模态 ：multimodal(title)；分类 cs.CL、cs.AI

Comments 18 pages, 9 figures, 12 tables

Enhancing Multimodal Understanding with CLIP-Based Image-to-Text Transformation

Chang Che, Qunwei Lin, Xinyu Zhao, Jiaxin Huang, Liqiang Yu

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

Cross-Modal Concept Learning and Inference for Vision-Language Models

Yi Zhang, Ce Zhang, Yushun Tang, Zhihai He

专题命中 图文多模态 ：cross-modal(title)；分类 cs.CV、cs.CL

Self-supervised Multi-modal Training from Uncurated Image and Reports Enables Zero-shot Oversight Artificial Intelligence in Radiology

Sangjoon Park, Eun Sun Lee, Kyung Sook Shin, Jeong Eun Lee, Jong Chul Ye

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.CL

Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding

Haoli Bai, Zhiguang Liu, Xiaojun Meng, Wentao Li, Shuang Liu, Nian Xie, Rongfu Zheng, Liangwei Wang, Lu Hou, Jiansheng Wei, Xin Jiang, Qun Liu

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.CL

Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions

Michele Cafagna, Kees van Deemter, Albert Gatt

专题命中 图文多模态 ：cross-modal(title)；分类 cs.CV、cs.CL

CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classification

Huidong Liu, Shaoyuan Xu, Jinmiao Fu, Yang Liu, Ning Xie, Chien-Chih Wang, Bryan Wang, Yi Sun

专题命中 图文多模态 ：image-text(title)；分类 cs.CV、cs.CL

Comments 9 pages, 2 figures, 6 tables, 1 algorithm

Dual-Path Convolutional Image-Text Embeddings with Instance Loss

Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, Mingliang Xu, Yi-Dong Shen

专题命中 图文多模态 ：image-text(title)；分类 cs.CV、cs.MM

Comments 15pages, 15 figures, 8 tables

OmniNet: A unified architecture for multi-modal multi-task learning

Subhojeet Pramanik, Priyanka Agrawal, Aman Hussain

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.CL

Comments Source code available at: https://github.com/subho406/OmniNet

Learn from Your Neighbor: Learning Multi-modal Mappings from Sparse Annotations

Ashwin Kalyan, Stefan Lee, Anitha Kannan, Dhruv Batra

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.CL

Comments To be presented at ICML 2018; 10 pages 5 figures

GuessWhat?! Visual object discovery through multi-modal dialogue

Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, Aaron Courville

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.AI

Comments 23 pages; CVPR 2017 submission; see https://guesswhat.ai

Vision-Language Grounding as Bidirectional Concept Correspondence

视觉-语言 Grounding 作为双向概念对应

Jieyu Zhang, Ziqi Gao, Luke Zettlemoyer, Ranjay Krishna

机构 * University of Washington（华盛顿大学） ； Allen Institute for AI（艾伦人工智能研究所） ； FAIR at Meta（Meta FAIR实验室） 纠错

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结 本研究将视觉-语言 Grounding 建模为双向概念对应，提出 ConCor-1 模型统一相关任务，在长文本数据集和零样本 LVIS 上对应 F1 分别提升 48%、29%，性能优于基线。

Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations

通过LENS：视觉语言模型表示的局部几何分解

Shalom Kachko, Raz Lapid, Margarita Vald, Almog Dubin, Moshe Sipper

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结 本研究提出LENS方法，将VLM激活分解为局部低秩高斯邻域，揭示LLaVA与Qwen3-VL的不同模态融合轨迹，该方法可实现因果生成干预并提升多模态检索性能。

An Exam for Active Observers

主动观察者的测试

Jiarui Zhang, Muzi Tao, Shangshang Wang, Ollie Liu, Xuezhe Ma, Willie Neiswanger

机构 * University of Southern California（南加州大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；MLLM(abstract_cn)；分类 cs.CV、cs.CL、cs.AI

AI总结 研究多模态大语言模型是否具备主动观察能力，引入ActiveVision基准测试，发现前沿模型表现不佳，即便能编写视觉代码差距仍存，表明当前模型缺乏主动视觉观察，呼吁构建闭合感知 - 推理循环的架构和训练目标。

Comments 17 pages, 8 figures, 4 tables. Project page: https://activevision.dev

SpatialThinker: Reinforcing Scene Graph-Grounded Spatial Reasoning via Dense Rewards

空间思考者：通过密集奖励强化场景图基础的空间推理

Hunar Batra, Haoqin Tu, Hardy Chen, Yuanze Lin, Cihang Xie, Ronald Clark

机构 * University of Oxford（牛津大学） ； University of California, Santa Cruz（加州大学圣克鲁兹分校） 纠错

专题命中 图文多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结 研究针对多模态大语言模型空间推理难题，提出SpatialThinker，通过在线强化学习统一场景图生成与视觉推理，构建心理场景图并借助密集奖励推理，贡献包括基于SGG推理、高质量训练数据集及密集奖励设计。

Comments Preprint. Accepted at NeurIPS 2025 Workshops on SPACE in Vision, Language, and Embodied AI (SpaVLE) as Oral, Embodied World Models for Decision Making (EWM), Aligning Reinforcement Learning Experimentalists and Theorists (ARLET), and Scaling Environments for Agents (SEA)