多模态大模型

SingaKids: A Multilingual Multimodal Dialogic Tutor for Language Learning

Zhengyuan Liu, Geyu Lin, Hui Li Tan, Huayun Zhang, Yanfeng Lu, Xiaoxue Gao, Stella Xin Yin, He Sun, Hock Huan Goh, Lung Hsiang Wong, Nancy F. Chen

机构 * Nanyang Technological University, Singapore（新加坡南洋理工大学） ； National Institute of Education (NIE), Singapore（新加坡国家教育研究所） ； Institute for Infocomm Research (I 2 R), A*STAR, Singapore（新加坡资讯与通信研究院（I2R）） 纠错

专题命中 图文多模态 ：multimodal(title)；分类 cs.CL、cs.AI

Comments ACL 2025 Industry Track

QARI-OCR: High-Fidelity Arabic Text Recognition through Multimodal Large Language Model Adaptation

Ahmed Wasfy, Omer Nacar, Abdelakreem Elkhateb, Mahmoud Reda, Omar Elshehy, Adel Ammar, Wadii Boulila

机构 * NAMAA ； KAND CA Corp. ； Prince Sultan University 纠错

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding

Yeongjae Cho, Keonwoo Kim, Taebaek Hwang, Sungzoon Cho

机构 * Seoul National University（首尔国立大学） ； Kim & Chang AI&IT System Center（金·昌AI&IT系统中心） ； Waddle Corporation（Waddle公司） 纠错

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

A Multimodal Pipeline for Clinical Data Extraction: Applying Vision-Language Models to Scans of Transfusion Reaction Reports

Henning Schäfer, Cynthia S. Schmidt, Johannes Wutzkowsky, Kamil Lorek, Lea Reinartz, Johannes Rückert, Christian Temme, Britta Böckmann, Peter A. Horn, Christoph M. Friedrich

机构 * Institute for Transfusion Medicine, University Hospital Essen, Hufelandstraße 55, Essen, Germany（1 输血医学研究所，埃森大学医院，Hufelandstraße 55，德国） ； Department of Computer Science, University of Applied Sciences and Arts Dortmund (FHDO), Emil-Figge Str. 42, Dortmund, Germany（2 计算机科学系，多特蒙德应用科学大学（FHDO），Emil-Figge Str. 42，德国） ； Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), University Hospital Essen, Hufelandstraße 55, Essen, Germany（3 医学信息学、生物统计学和流行病学研究所（IMIBE），埃森大学医院，Hufelandstraße 55，德国） ； Institute for AI in Medicine (IKIM), University Hospital Essen, Girardetstraße 2, Essen, Germany（4 医学人工智能研究所（IKIM），埃森大学医院，Girardetstraße 2，德国） ； Institute of Interventional and Diagnostic Radiology and Neuroradiology, University Hospital Essen, Hufelandstraße 55, Essen, Germany（5 干预性和诊断放射学及神经放射学研究所，埃森大学医院，Hufelandstraße 55，德国） ； Department of Dermatology, University Hospital Essen, Hufelandstraße 55, Essen, Germany（6 皮肤科系，埃森大学医院，Hufelandstraße 55，德国） 纠错

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.CL

GeoRSMLLM: A Multimodal Large Language Model for Vision-Language Tasks in Geoscience and Remote Sensing

Zilun Zhang, Haozhan Shen, Tiancheng Zhao, Bin Chen, Zian Guan, Yuhao Wang, Xu Jia, Yuxiang Cai, Yongheng Shang, Jianwei Yin

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

Multimodal Cognitive Reframing Therapy via Multi-hop Psychotherapeutic Reasoning

Subin Kim, Hoonrae Kim, Heejin Do, Gary Geunbae Lee

专题命中 图文多模态 ：multimodal(title)；分类 cs.CL、cs.AI

Comments NAACL 2025 Main

A Multi-Modal Explainability Approach for Human-Aware Robots in Multi-Party Conversation

Iveta Bečková, Štefan Pócoš, Giulia Belgiovine, Marco Matarese, Omar Eldardeer, Alessandra Sciutti, Carlo Mazzola

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CL、cs.AI

Comments 32pp (+6pp sup.mat.) Accepted in Computer Vision and Image Understanding Journal on January 23, 2025. This research received funding Horizon-Europe TERAIS project (G.A. 101079338) and Slovak Research and Development Agency, project no. APVV-21-0105

MM-GEN: Enhancing Task Performance Through Targeted Multimodal Data Curation

Siddharth Joshi, Besmira Nushi, Vidhisha Balachandran, Varun Chandrasekaran, Vibhav Vineet, Neel Joshi, Baharan Mirzasoleiman

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.CL

Clustering-based Image-Text Graph Matching for Domain Generalization

Nokyung Park, Daewon Chae, Jeongyong Shim, Sangpil Kim, Eun-Sol Kim, Jinkyu Kim

专题命中 图文多模态 ：image-text(title)；分类 cs.CV、cs.AI

RoCOCO: Robustness Benchmark of MS-COCO to Stress-test Image-Text Matching Models

Seulki Park, Daeho Um, Hajung Yoon, Sanghyuk Chun, Sangdoo Yun, Jin Young Choi

专题命中 图文多模态 ：image-text(title)；分类 cs.CV、cs.AI

Comments Accepted to ECCV Synthetic Data for Computer Vision Workshop (Oral)

Vision Search Assistant: Empower Vision-Language Models as Multimodal Search Engines

Zhixin Zhang, Yiyuan Zhang, Xiaohan Ding, Xiangyu Yue

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

Comments Code is available at https://github.com/cnzzx/VSA

HeGraphAdapter: Tuning Multi-Modal Vision-Language Models with Heterogeneous Graph Adapter

Yumiao Zhao, Bo Jiang, Xiao Wang, Qin Xu, Jin Tang

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.MM

FairPIVARA: Reducing and Assessing Biases in CLIP-Based Multimodal Models

Diego A. B. Moreira, Alef Iury Ferreira, Jhessica Silva, Gabriel Oliveira dos Santos, Luiz Pereira, João Medrado Gondim, Gustavo Bonil, Helena Maia, Nádia da Silva, Simone Tiemi Hashiguti, Jefersson A. dos Santos, Helio Pedrini, Sandra Avila

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

Comments 14 pages, 10 figures. Accepted to 35th British Machine Vision Conference (BMVC 2024), Workshop on Privacy, Fairness, Accountability and Transparency in Computer Vision

BCAmirs at SemEval-2024 Task 4: Beyond Words: A Multimodal and Multilingual Exploration of Persuasion in Memes

Amirhossein Abaskohi, Amirhossein Dabiriaghdam, Lele Wang, Giuseppe Carenini

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.CL

Comments 12 pages, 5 tables, 2 figures, Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024) @ NAACL 2024

A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interaction

Yu Hao, Fan Yang, Hao Huang, Shuaihang Yuan, Sundeep Rangan, John-Ross Rizzo, Yao Wang, Yi Fang

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.AI

Let's Go Shopping (LGS) -- Web-Scale Image-Text Dataset for Visual Concept Understanding

Yatong Bai, Utsav Garg, Apaar Shanker, Haoming Zhang, Samyak Parajuli, Erhan Bas, Isidora Filipovic, Amelia N. Chu, Eugenia D Fomitcheva, Elliot Branson, Aerin Kim, Somayeh Sojoudi, Kyunghyun Cho

专题命中 图文多模态 ：image-text(title)；分类 cs.CV、cs.AI

Self-Imagine: Effective Unimodal Reasoning with Multimodal Models using Self-Imagination

Syeda Nahida Akter, Aman Madaan, Sangwu Lee, Yiming Yang, Eric Nyberg

专题命中 图文多模态 ：multimodal(title)；分类 cs.CL、cs.AI

Comments 18 pages, 9 figures, 12 tables

Enhancing Multimodal Understanding with CLIP-Based Image-to-Text Transformation

Chang Che, Qunwei Lin, Xinyu Zhao, Jiaxin Huang, Liqiang Yu

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV、cs.AI

Cross-Modal Concept Learning and Inference for Vision-Language Models

Yi Zhang, Ce Zhang, Yushun Tang, Zhihai He

专题命中 图文多模态 ：cross-modal(title)；分类 cs.CV、cs.CL

Self-supervised Multi-modal Training from Uncurated Image and Reports Enables Zero-shot Oversight Artificial Intelligence in Radiology

Sangjoon Park, Eun Sun Lee, Kyung Sook Shin, Jeong Eun Lee, Jong Chul Ye

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.CL

Wukong-Reader: Multi-modal Pre-training for Fine-grained Visual Document Understanding

Haoli Bai, Zhiguang Liu, Xiaojun Meng, Wentao Li, Shuang Liu, Nian Xie, Rongfu Zheng, Liangwei Wang, Lu Hou, Jiansheng Wei, Xin Jiang, Qun Liu

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.CL

Understanding Cross-modal Interactions in V&L Models that Generate Scene Descriptions

Michele Cafagna, Kees van Deemter, Albert Gatt

专题命中 图文多模态 ：cross-modal(title)；分类 cs.CV、cs.CL

CMA-CLIP: Cross-Modality Attention CLIP for Image-Text Classification

Huidong Liu, Shaoyuan Xu, Jinmiao Fu, Yang Liu, Ning Xie, Chien-Chih Wang, Bryan Wang, Yi Sun

专题命中 图文多模态 ：image-text(title)；分类 cs.CV、cs.CL

Comments 9 pages, 2 figures, 6 tables, 1 algorithm

Dual-Path Convolutional Image-Text Embeddings with Instance Loss

Zhedong Zheng, Liang Zheng, Michael Garrett, Yi Yang, Mingliang Xu, Yi-Dong Shen

专题命中 图文多模态 ：image-text(title)；分类 cs.CV、cs.MM

Comments 15pages, 15 figures, 8 tables

OmniNet: A unified architecture for multi-modal multi-task learning

Subhojeet Pramanik, Priyanka Agrawal, Aman Hussain

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.CL

Comments Source code available at: https://github.com/subho406/OmniNet

Learn from Your Neighbor: Learning Multi-modal Mappings from Sparse Annotations

Ashwin Kalyan, Stefan Lee, Anitha Kannan, Dhruv Batra

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.CL

Comments To be presented at ICML 2018; 10 pages 5 figures

GuessWhat?! Visual object discovery through multi-modal dialogue

Harm de Vries, Florian Strub, Sarath Chandar, Olivier Pietquin, Hugo Larochelle, Aaron Courville

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV、cs.AI

Comments 23 pages; CVPR 2017 submission; see https://guesswhat.ai

From Inference to Adaptation: A Unified Optimal Transport View of Vision Language Model

从推理到适应：视觉语言模型的统一最优传输视角

Qi Yu, Zhichen Zeng, Katherine Tieu, Xiyuan Yang, Ruizhong Qiu, Yuchen Yan, Lihui Liu, Yanjun Zhao, Lingjie Chen, Jingrui He, Hanghang Tong

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） ； Amazon（亚马逊公司） ； Wayne State University（韦恩州立大学） 纠错

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结 本研究提出名为algname的VLM测试时适应方法，通过Wasserstein最优传输统一VLM的推理与适应目标，实验显示其性能较最优方法提升7%且效率先进。

Vision-Language Grounding as Bidirectional Concept Correspondence

视觉-语言 Grounding 作为双向概念对应

Jieyu Zhang, Ziqi Gao, Luke Zettlemoyer, Ranjay Krishna

机构 * University of Washington（华盛顿大学） ； Allen Institute for AI（艾伦人工智能研究所） ； FAIR at Meta（Meta FAIR实验室） 纠错

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结 本研究将视觉-语言 Grounding 建模为双向概念对应，提出 ConCor-1 模型统一相关任务，在长文本数据集和零样本 LVIS 上对应 F1 分别提升 48%、29%，性能优于基线。

Through the LENS: Local Geometric Decomposition of Vision-Language Model Representations

通过LENS：视觉语言模型表示的局部几何分解

Shalom Kachko, Raz Lapid, Margarita Vald, Almog Dubin, Moshe Sipper

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结 本研究提出LENS方法，将VLM激活分解为局部低秩高斯邻域，揭示LLaVA与Qwen3-VL的不同模态融合轨迹，该方法可实现因果生成干预并提升多模态检索性能。