多模态大模型 - arXivDaily 专题

2106.11360 2021-06-23 cs.LG 78%

Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records

Yikuan Li, Mohammad Mamouei, Gholamreza Salimi-Khorshidi, Shishir Rao, Abdelaali Hassaine, Dexter Canoy, Thomas Lukasiewicz, Kazem Rahimi

专题命中跨模态检索：multimodal(title,abstract)

2007.01179 2021-04-22 cs.LG stat.ML 78%

Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models

Yuge Shi, Brooks Paige, Philip H. S. Torr, N. Siddharth

专题命中跨模态检索：multimodal(title,abstract)

2101.03525 2021-03-02 cs.RO cs.LG 78%

Cross-Modal Contrastive Learning of Representations for Navigation using Lightweight, Low-Cost Millimeter Wave Radar for Adverse Environmental Conditions

Jui-Te Huang, Chen-Lung Lu, Po-Kai Chang, Ching-I Huang, Chao-Chun Hsu, Zu Lin Ewe, Po-Jui Huang, Hsueh-Cheng Wang

专题命中跨模态检索：cross-modal(title,abstract)

Comments For further details, please visit https://arg-nctu.github.io/projects/deeprl-mmWave.html

Journal ref IEEE Robotics and Automation Letters, 2021

2010.01666 2020-10-06 cs.IR cs.LG 78%

Multi-Modal Retrieval using Graph Neural Networks

Aashish Kumar Misraa, Ajinkya Kale, Pranav Aggarwal, Ali Aminian

专题命中跨模态检索：multi-modal(title,abstract)

2007.03634 2020-07-08 cs.LG cs.IR cs.SI stat.ML 78%

PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest

Aditya Pal, Chantat Eksombatchai, Yitong Zhou, Bo Zhao, Charles Rosenberg, Jure Leskovec

专题命中跨模态检索：multi-modal(title,abstract)

Comments 10 pages, 7 figures

Journal ref KDD 2020

2003.12164 2020-04-09 cs.RO 78%

Representing Multi-Robot Structure through Multimodal Graph Embedding for the Selection of Robot Teams

Brian Reily, Christopher Reardon, Hao Zhang

专题命中跨模态检索：multimodal(title,abstract)

Comments Accepted to International Conference on Robotics and Automation (ICRA) 2020, IEEE copyright

1909.13516 2019-10-01 cs.SE cs.PL 78%

Multi-Modal Attention Network Learning for Semantic Source Code Retrieval

Yao Wan, Jingdong Shu, Yulei Sui, Guandong Xu, Zhou Zhao, Jian Wu, Philip S. Yu

专题命中跨模态检索：multi-modal(title,abstract)

1904.02449 2019-04-05 cs.IR 78%

Triplet-Based Deep Hashing Network for Cross-Modal Retrieval

Cheng Deng, Zhaojia Chen, Xianglong Liu, Xinbo Gao, Dacheng Tao

专题命中跨模态检索：cross-modal(title,abstract)

1903.12019 2019-03-29 cs.LG stat.ML 78%

Multimodal Deep Network Embedding with Integrated Structure and Attribute Information

Conghui Zheng, Li Pan, Peng Wu

专题命中跨模态检索：multimodal(title,abstract)

Comments 15 pages, 10 figures

1903.02149 2019-03-07 cs.IR 78%

Coupled CycleGAN: Unsupervised Hashing Network for Cross-Modal Retrieval

Chao Li, Cheng Deng, Lei Wang, De Xie, Xianglong Liu

专题命中跨模态检索：cross-modal(title,abstract)

1811.11422 2018-11-29 cs.IR 78%

Exploiting "Quantum-like Interference" in Decision Fusion for Ranking Multimodal Documents

Dimitris Gkoumas, Dawei Sogn

专题命中跨模态检索：multimodal(title,abstract)

1802.09729 2018-07-27 cs.IR cs.LG cs.SE 78%

Network-Clustered Multi-Modal Bug Localization

Thong Hoang, Richard J. Oentaryo, Tien-Duy B. Le, David Lo

专题命中跨模态检索：multi-modal(title,abstract)

Comments IEEE Transactions on Software Engineering

1805.12312 2018-06-01 cs.IR 78%

Collaborative Multi-modal deep learning for the personalized product retrieval in Facebook Marketplace

Lu Zheng, Zhao Tan, Kun Han, Ren Mao

专题命中跨模态检索：multi-modal(title,abstract)

Comments 6 pages, 4 figures

1802.00985 2018-02-14 cs.IR 78%

Modeling Text with Graph Convolutional Network for Cross-Modal Information Retrieval

Jing Yu, Yuhang Lu, Zengchang Qin, Yanbing Liu, Jianlong Tan, Li Guo, Weifeng Zhang

专题命中跨模态检索：cross-modal(title,abstract)

Comments 7 pages, 11 figures

1801.02693 2018-01-10 cs.MA cs.DS 78%

Stable Marriage with Multi-Modal Preferences

Jiehua Chen, Rolf Niedermeier, Piotr Skowron

专题命中跨模态检索：multi-modal(title,abstract)

1707.08322 2017-07-27 cs.IR 78%

Discrete Latent Factor Model for Cross-Modal Hashing

Qing-Yuan Jiang, Wu-Jun Li

专题命中跨模态检索：cross-modal(title,abstract)

1608.00134 2017-02-23 cs.IR cs.DB 78%

A Graph Framework for Multimodal Medical Information Processing

Georgios Drakopoulos, Vasileios Megalooikonomou

专题命中跨模态检索：multimodal(title,abstract)

Comments We need to correct certain errors both in the software description as well as in the algorithms

1602.02255 2016-02-16 cs.IR 78%

Deep Cross-Modal Hashing

Qing-Yuan Jiang, Wu-Jun Li

专题命中跨模态检索：cross-modal(title,abstract)

Comments 12 pages

1504.02265 2015-04-13 q-bio.NC nlin.AO physics.soc-ph 78%

An Algebraic Topological Method for Multimodal Brain Networks Comparisons

Tiago Simas, Mario Chavez, Pablo Rodriguez, Albert Diaz-Guilera

专题命中跨模态检索：multimodal(title,abstract)

1503.07906 2015-03-30 cs.LG stat.ML 78%

Generalized K-fan Multimodal Deep Model with Shared Representations

Gang Chen, Sargur N. Srihari

专题命中跨模态检索：multimodal(title,abstract)

Comments 11 pages, 5 figures

2608.12209 2026-08-13 cs.CV 新提交 77%

Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction

作为辅助监督的生成：通过解耦嵌入预测以零推理开销增强视觉理解

Zhongbin Guo, Jiahao Xie, Dongling Xiao, Qianle Wang, Ruiqi Lu, Xiaomin He, Wanxuan Sun, Cheng Yang

机构 * ByteDance（字节跳动）

专题命中跨模态检索：multimodal(abstract,abstract_cn)；cross-modal(abstract)；分类 cs.CV

AI总结本研究提出GAS框架，将视觉生成作为辅助监督，通过解耦MoT架构的NEP实现零推理开销，提升了多模态理解尤其是感知与空间理解能力。

2607.20291 2026-07-23 cs.CV 新提交 77%

Diverse-Intent Multi-Turn Fashion Image Retrieval

多样化意图的多轮时尚图像检索

Mingqiang Tang, Haokun Wen, Meng Liu, Yupeng Hu, Weili Guan, Xuemeng Song

专题命中跨模态检索：MLLM(abstract,abstract_cn)；multimodal(abstract)；分类 cs.CV

AI总结研究现实世界时尚多轮图像检索问题，提出FashionAM框架直接对齐多模态对话查询与时尚图库嵌入空间，避免文本化，引入DIM-Fashion数据集，实验证明该框架优于现有方法，数据集和代码将公开。

2607.05901 2026-07-08 cs.AI 新提交 77%

Uncovering Latent Depression Severity for Binary Depression Detection via Advantage-weighting Ranking

通过优势加权排序揭示二元抑郁检测中的潜在抑郁严重程度

Manning Gao, Tingyi Liu, Leheng Zhang, Haifeng Hu, Yuncheng Jiang, Sijie Mai

机构 * South China Normal University（华南师范大学）

专题命中跨模态检索：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.AI

AI总结研究利用视听数据检测抑郁的挑战，提出含时间编码器和相互变压器的多模态框架，核心是二元优势加权排序损失，通过两种机制优化潜在空间分布，实验表明该模型重建潜在有序结构，性能达最优。

2607.00060 2026-07-02 cs.CV 新提交 77%

Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence

协同感知-推理治理：用可验证解剖证据为医学多模态大语言模型提供基础

Rui Hao, Qiankun Li, Junyuan Mao, Linghao Meng, Dirui Xie, Dayu Tan, Zhigang Zeng

机构 * Huazhong University of Science and Technology（华中科技大学）； Imperial Global Singapore, Imperial College London（帝国理工学院新加坡全球中心）； Nanyang Technological University（南洋理工大学）； National University of Singapore（新加坡国立大学）； Anhui University（安徽大学）

专题命中跨模态检索：MLLM(summary_cn)；multimodal(abstract)；分类 cs.CV

AI总结提出一种无需训练的证据注入框架，通过ROI引导的视觉激活调制和解剖坐标语义标记，协同校准视觉感知与文本推理，动态路由任务特定干预，有效减少医学MLLM的幻觉。

Comments Accepted by MICCAI 2026 (Early Accept, Top 9%)

2507.00263 2026-07-01 cs.CV cs.LG cs.NE 版本更新 77%

Room Scene Discovery and Grouping in Unstructured Vacation Rental Image Collections

非结构化度假租赁图像集合中的房间场景发现与分组

Vignesh Ram Nithin Kappagantula, Shayan Hassantabar

机构 * Expedia Group（Expedia集团）

专题命中跨模态检索：MLLM(abstract,abstract_cn)；multi-modal(abstract)；分类 cs.CV

AI总结针对度假租赁平台图片缺乏结构化分类的问题，提出一种低延迟、样本高效的机器学习流水线，通过监督式房间类型检测、重叠检测和聚类算法实现房间分组，并利用多模态大语言模型识别床型，显著优于对比学习和预训练嵌入聚类方法。

Comments Presented at the Two-sided Marketplace Optimization Workshop, KDD 2025

2606.23539 2026-06-23 cs.CV 新提交 77%

LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement

LightSTAR: 通过视觉自适应精炼的轻量级选择实现高效视觉文档检索

Tongkun Guan, Haocheng Wang, Wei Shen, Xiaokang Yang

机构 * MoE Key Lab of Artificial Intelligence, AI Institute, School of Computer Science, Shanghai Jiao Tong University（上海交通大学计算机科学与工程学院人工智能研究院教育部人工智能重点实验室）

专题命中跨模态检索：MLLM(abstract,abstract_cn)；multi-modal(abstract)；分类 cs.CV

AI总结提出LightSTAR框架，通过免LLM的视觉选择快速筛选候选页，再经视觉自适应语义精炼进行细粒度匹配，在保持高检索精度的同时大幅降低延迟。

Comments Accpeted by ECCV 2026

2510.08976 2026-06-16 cs.CV cs.DC cs.IR 版本更新 77%

MIRAGE: Runtime Scheduling for Multi-Vector Image Retrieval with Hierarchical Decomposition

MIRAGE：基于层次分解的多向量图像检索运行时调度

Maoliang Li, Ke Li, Yaoyang Liu, Jiayu Chen, Zihao Zheng, Yinjun Wu, Chenchen Liu, Xiang Chen

机构 * School of Computer Science, Peking University（北京大学计算机科学学院）； School of Electronics Engineering and Computer Science, Peking University（北京大学电子工程与计算机科学学院）； School of Information, Renmin University of China（中国人民大学信息学院）； School of Integrated Circuit Science and Engineering, Beihang University（北京航空航天大学集成电路科学与工程学院）

专题命中跨模态检索：MLLM(abstract,abstract_cn)；multimodal(abstract)；分类 cs.CV

AI总结提出MIRAGE框架，通过层次化分解和跨层次相似性一致性减少冗余计算，实现多向量图像检索的精度提升和3.5倍计算加速。

Comments Will appear in DAC'2026, camera ready

2606.12294 2026-06-11 cs.CV eess.IV 新提交 77%

Bridging the Modality Gap in Forensic Image Retrieval

弥合法医图像检索中的模态差距

Ricardo González-Gazapo, Annette Morales-González, Yoanna Martínez-Díaz, Heydi Méndez-Vázquez, Milton García-Borroto

机构 * Advanced Technologies Application Center (CENATAV)（先进技术应用中心（CENATAV））； Centro de Sistemas Complejos, Facultad de Física, Universidad de La Habana（哈瓦那大学物理学院复杂系统中心）

专题命中跨模态检索：MLLM(abstract,abstract_cn)；multimodal(abstract)；分类 cs.CV

AI总结提出统一检索框架，利用多模态大语言模型生成文本描述并结合视觉与文本特征融合，提升纹身、人脸素描等法医任务的检索精度与鲁棒性。

Comments 23 pages, 5 figures, paper submitted to Elsevier journal

2606.03470 2026-06-03 cs.CV 77%

Mixed-Modality Dual Face-Hair Retrieval

混合模态双人脸-发型检索

Quoc-Anh Bui-Huynh, Mai-Tuyen Lam, Dai-Anh-Tuan Nguyen, Thanh Duc Ngo

机构 * Vietnam National University, Ho Chi Minh City, Vietnam（越南国家大学，胡志明市，越南）； University of Information Technology, VNU-HCM, Ho Chi Minh City, Vietnam（信息技术大学，VNU-HCM，胡志明市，越南）

专题命中跨模态检索：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

AI总结提出混合模态双参考检索任务DFHR，通过解耦身份与发型特征并融合多模态嵌入，实现跨模态的身份感知与属性可控检索。

2604.06484 2026-06-01 cs.CL 77%

ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs

ValueGround: 评估多模态大语言模型中文化条件化的视觉价值基础

Zhipin Wang, Christoph Leiter, Christian Frey, Mohamed Hesham Ibrahim Abdalla, Josif Grabocka, Steffen Eger

机构 * University of Technology Nuremberg (UTN)（图恩大学）

专题命中跨模态检索：multimodal(abstract)；MLLM(abstract_cn)；cross-modal(abstract)；分类 cs.CL

AI总结提出ValueGround基准，通过最小对比图像对评估多模态大语言模型在文化条件化视觉价值判断中的表现，发现模型在可视化选项下准确率显著低于文本选项。

Comments Updated preprint

1. 跨模态检索 3454 篇

Hi-BEHRT: Hierarchical Transformer-based model for accurate prediction of clinical events using multimodal longitudinal electronic health records

Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models

Cross-Modal Contrastive Learning of Representations for Navigation using Lightweight, Low-Cost Millimeter Wave Radar for Adverse Environmental Conditions

Multi-Modal Retrieval using Graph Neural Networks

PinnerSage: Multi-Modal User Embedding Framework for Recommendations at Pinterest

Representing Multi-Robot Structure through Multimodal Graph Embedding for the Selection of Robot Teams

Multi-Modal Attention Network Learning for Semantic Source Code Retrieval

Triplet-Based Deep Hashing Network for Cross-Modal Retrieval

Multimodal Deep Network Embedding with Integrated Structure and Attribute Information

Coupled CycleGAN: Unsupervised Hashing Network for Cross-Modal Retrieval

Exploiting "Quantum-like Interference" in Decision Fusion for Ranking Multimodal Documents

Network-Clustered Multi-Modal Bug Localization

Collaborative Multi-modal deep learning for the personalized product retrieval in Facebook Marketplace

Modeling Text with Graph Convolutional Network for Cross-Modal Information Retrieval

Stable Marriage with Multi-Modal Preferences

Discrete Latent Factor Model for Cross-Modal Hashing

A Graph Framework for Multimodal Medical Information Processing

Deep Cross-Modal Hashing

An Algebraic Topological Method for Multimodal Brain Networks Comparisons

Generalized K-fan Multimodal Deep Model with Shared Representations

Generation as Auxiliary Supervision: Enhancing Visual Understanding at Zero Inference Overhead via Decoupled Embedding Prediction

Diverse-Intent Multi-Turn Fashion Image Retrieval

Uncovering Latent Depression Severity for Binary Depression Detection via Advantage-weighting Ranking

Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence

Room Scene Discovery and Grouping in Unstructured Vacation Rental Image Collections

LightSTAR: Efficient Visual Document Retrieval via Lightweight Selection with Vision-Adaptive Refinement

MIRAGE: Runtime Scheduling for Multi-Vector Image Retrieval with Hierarchical Decomposition

Bridging the Modality Gap in Forensic Image Retrieval

Mixed-Modality Dual Face-Hair Retrieval

ValueGround: Evaluating Culture-Conditioned Visual Value Grounding in MLLMs