多模态大模型 - arXivDaily 专题

2504.03184 2025-06-25 cs.IR 82%

Answering Multimodal Exclusion Queries with Lightweight Sparse Disentangled Representations

Prachi J, Sumit Bhatia, Srikanta Bedathur

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

Comments In Proceedings of the 2025 International ACM SIGIR Conference on Innovative Concepts and Theories in Information Retrieval (ICTIR)

2506.12364 2025-06-24 cs.AI cs.CL cs.CV 82%

MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval

Mingjun Xu, Jinhan Dong, Jue Hou, Zehui Wang, Sihang Li, Zhifeng Gao, Renxin Zhong, Hengxing Cai

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2311.06456 2025-05-15 cs.LG 82%

Advancing Drug Discovery with Enhanced Chemical Understanding via Asymmetric Contrastive Multimodal Learning

Yifei Wang, Yunrui Li, Lin Liu, Pengyu Hong, Hao Xu

机构 * Brandeis University（布兰迪斯大学）； Stanford University（斯坦福大学）； Northeastern University（东北大学）； Harvard Medical School（哈佛医学院）

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

Comments 51 pages, 19 figures, 11 tables

2412.13509 2025-03-26 cs.HC 82%

Visualizing the Invisible: A Generative AR System for Intuitive Multi-Modal Sensor Data Presentation

Yunqi Guo, Kaiyuan Hou, Heming Fu, Hongkai Chen, Zhenyu Yan, Guoliang Xing, Xiaofan Jiang

专题命中跨模态检索：multi-modal(title,abstract)；cross-modal(abstract)

2407.19711 2025-03-24 cs.SE 82%

TVDiag: A Task-oriented and View-invariant Failure Diagnosis Framework with Multimodal Data

Shuaiyu Xie, Jian Wang, Hanbin He, Zhihao Wang, Yuqi Zhao, Neng Zhang, Bing Li

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

Comments 32 pages

2503.16064 2025-03-21 cs.CV cs.AI cs.IR cs.MM 82%

PromptHash: Affinity-Prompted Collaborative Cross-Modal Learning for Adaptive Hashing Retrieval

Qiang Zou, Shuli Cheng, Jiayi Chen

专题命中跨模态检索：cross-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by CVPR2025

2503.04252 2025-03-07 cs.DB cs.LG 82%

RCRank: Multimodal Ranking of Root Causes of Slow Queries in Cloud Database Systems

Biao Ouyang, Yingying Zhang, Hanyin Cheng, Yang Shu, Chenjuan Guo, Bin Yang, Qingsong Wen, Lunting Fan, Christian S. Jensen

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

Comments Accepted by VLDB 2025

2411.12126 2025-03-06 cs.LG 82%

MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT

Xiaomin Ouyang, Jason Wu, Tomoyoshi Kimura, Yihan Lin, Gunjan Verma, Tarek Abdelzaher, Mani Srivastava

专题命中跨模态检索：multimodal(title,abstract)；multimodal foundation model(abstract)

2409.07402 2025-03-06 cs.LG cs.AI cs.CL cs.CV 82%

What to align in multimodal contrastive learning?

Benoit Dufumier, Javiera Castillo-Navarro, Devis Tuia, Jean-Philippe Thiran

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICLR 2025, 25 pages

2502.18763 2025-02-27 cs.IT math.IT 82%

CommGPT: A Graph and Retrieval-Augmented Multimodal Communication Foundation Model

Feibo Jiang, Wanyun Zhu, Li Dong, Kezhi Wang, Kun Yang, Cunhua Pan, Octavia A. Dobre

专题命中跨模态检索：multimodal(title,abstract)；multimodal foundation model(abstract)

2501.17799 2025-02-18 cs.HC cs.IR 82%

Leveraging Multimodal LLM for Inspirational User Interface Search

Seokhyeon Park, Yumin Song, Soohyun Lee, Jaeyoung Kim, Jinwook Seo

专题命中跨模态检索：multimodal(title,abstract)；MLLM(abstract)

Comments In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '25)

2502.00653 2025-02-04 cs.CR 82%

Towards Robust Multimodal Large Language Models Against Jailbreak Attacks

Ziyi Yin, Yuanpu Cao, Han Liu, Ting Wang, Jinghui Chen, Fenhlong Ma

专题命中跨模态检索：multimodal(title,abstract)；MLLM(abstract)

2410.05160 2025-01-03 cs.CV cs.AI cs.CL 82%

VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

Ziyan Jiang, Rui Meng, Xinyi Yang, Semih Yavuz, Yingbo Zhou, Wenhu Chen

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Technical Report

2412.14835 2024-12-20 cs.CL cs.AI cs.CV cs.IR 82%

Progressive Multimodal Reasoning via Active Retrieval

Guanting Dong, Chenghao Zhang, Mengjie Deng, Yutao Zhu, Zhicheng Dou, Ji-Rong Wen

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Working in progress

2409.12728 2024-12-19 q-bio.GN cs.LG 82%

PRAGA: Prototype-aware Graph Adaptive Aggregation for Spatial Multi-modal Omics Analysis

Xinlei Huang, Zhiqi Ma, Dian Meng, Yanran Liu, Shiwei Ruan, Qingqiang Sun, Xubin Zheng, Ziyue Qiao

专题命中跨模态检索：multi-modal(title,abstract)；cross-modal(abstract)

Comments Accepted by AAAl2025; full version including appendix

2411.16096 2024-11-26 cs.CV cs.AI cs.MM 82%

ENCLIP: Ensembling and Clustering-Based Contrastive Language-Image Pretraining for Fashion Multimodal Search with Limited Data and Low-Quality Images

Prithviraj Purushottam Naik, Rohit Agarwal

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2411.04952 2024-11-08 cs.CV cs.AI cs.CL 82%

M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

Jaemin Cho, Debanjan Mahata, Ozan Irsoy, Yujie He, Mohit Bansal

专题命中跨模态检索：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project webpage: https://m3docrag.github.io

2410.20482 2024-10-29 cs.CL cs.AI cs.CV 82%

What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration

Libo Qin, Qiguang Chen, Hao Fei, Zhi Chen, Min Li, Wanxiang Che

专题命中跨模态检索：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at NeurIPS 2024

2409.12295 2024-09-20 cs.LG stat.ML 82%

SANE: Strategic Autonomous Non-Smooth Exploration for Multiple Optima Discovery in Multi-modal and Non-differentiable Black-box Functions

Arpan Biswas, Rama Vasudevan, Rohit Pant, Ichiro Takeuchi, Hiroshi Funakubo, Yongtao Liu

专题命中跨模态检索：multi-modal(title,abstract)；multimodal(abstract)

Comments 25 pages, 7 figures in main text, 2 figures in Supp Mat

2408.00491 2024-08-02 cs.CL cs.CV cs.MM 82%

GalleryGPT: Analyzing Paintings with Large Multimodal Models

Yi Bin, Wenhao Shi, Yujuan Ding, Zhiqiang Hu, Zheng Wang, Yang Yang, See-Kiong Ng, Heng Tao Shen

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted as Oral Presentation at ACM Multimedia 2024

2407.20709 2024-07-31 cs.RO 82%

A Case Study on Visual-Audio-Tactile Cross-Modal Retrieval

Jagoda Wojcik, Jiaqi Jiang, Jiacheng Wu, Shan Luo

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)

Comments 7 pages, 6 figures, accepted to the 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2024)

2407.06219 2024-07-10 cs.NE 82%

In Search of Excellence: SHOA as a Competitive Shrike Optimization Algorithm for Multimodal Problems

Hanan K. AbdulKarim, Tarik A. Rashid

专题命中跨模态检索：multimodal(title,abstract)；multi-modal(abstract)

Comments 25 pages

2309.05423 2024-06-12 eess.AS cs.AI cs.CL cs.SD 82%

Multi-Modal Automatic Prosody Annotation with Contrastive Pretraining of SSWP

Jinzuomu Zhong, Yang Li, Hui Huang, Korin Richmond, Jie Liu, Zhiba Su, Jing Guo, Benlai Tang, Fengjie Zhu

专题命中跨模态检索：multi-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

2405.17777 2024-05-29 cs.IR 82%

RREH: Reconstruction Relations Embedded Hashing for Semi-Paired Cross-Modal Retrieval

Jianzong Wang, Haoxiang Shi, Kaiyi Luo, Xulong Zhang, Ning Cheng, Jing Xiao

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)

Comments Accepted by the 20th International Conference on Intelligent Computing (ICIC 2024)

2404.01958 2024-04-03 cs.LG 82%

MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels

Lilin Xu, Chaojie Gu, Rui Tan, Shibo He, Jiming Chen

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)

Comments Accepted to the 21th ACM Conference on Embedded Networked Sensor Systems (SenSys 2023)

2307.16847 2024-02-20 cs.LG 82%

CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Masking

Shohreh Deldari, Dimitris Spathis, Mohammad Malekzadeh, Fahim Kawsar, Flora Salim, Akhil Mathur

专题命中跨模态检索：cross-modal(title,abstract)；multimodal(abstract)

Comments Accepted in WSDM24. Short version presented in ML4MHD @ICML23

2305.01447 2024-01-31 cs.MM cs.CL cs.CV cs.DB cs.IR 82%

Multimodal Neural Databases

Giovanni Trappolini, Andrea Santilli, Emanuele Rodolà, Alon Halevy, Fabrizio Silvestri

专题命中跨模态检索：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Journal ref SIGIR 2023: Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval

2312.12094 2023-12-21 q-bio.BM 82%

CrossBind: Collaborative Cross-Modal Identification of Protein Nucleic-Acid-Binding Residues

Linglin Jing, Sheng Xu, Yifan Wang, Yuzhe Zhou, Tao Shen, Zhigang Ji, Hui Fang, Zhen Li, Siqi Sun

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)

Comments Accepted to AAAI-24

2312.07435 2023-12-13 cs.CV cs.AI cs.CL cs.LG 82%

Cross-modal Contrastive Learning with Asymmetric Co-attention Network for Video Moment Retrieval

Love Panta, Prashant Shrestha, Brabeem Sapkota, Amrita Bhattarai, Suresh Manandhar, Anand Kumar Sah

专题命中跨模态检索：cross-modal(title)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

2310.13267 2023-10-23 cs.CL cs.CV cs.LG cs.SD eess.AS 82%

On the Language Encoder of Contrastive Cross-modal Models

Mengjie Zhao, Junya Ono, Zhi Zhong, Chieh-Hsin Lai, Yuhta Takida, Naoki Murata, Wei-Hsiang Liao, Takashi Shibuya, Hiromi Wakaki, Yuki Mitsufuji

专题命中跨模态检索：cross-modal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

1. 跨模态检索 3437 篇

Answering Multimodal Exclusion Queries with Lightweight Sparse Disentangled Representations

MM-R5: MultiModal Reasoning-Enhanced ReRanker via Reinforcement Learning for Document Retrieval

Advancing Drug Discovery with Enhanced Chemical Understanding via Asymmetric Contrastive Multimodal Learning

Visualizing the Invisible: A Generative AR System for Intuitive Multi-Modal Sensor Data Presentation

TVDiag: A Task-oriented and View-invariant Failure Diagnosis Framework with Multimodal Data

PromptHash: Affinity-Prompted Collaborative Cross-Modal Learning for Adaptive Hashing Retrieval

RCRank: Multimodal Ranking of Root Causes of Slow Queries in Cloud Database Systems

MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoT

What to align in multimodal contrastive learning?

CommGPT: A Graph and Retrieval-Augmented Multimodal Communication Foundation Model

Leveraging Multimodal LLM for Inspirational User Interface Search

Towards Robust Multimodal Large Language Models Against Jailbreak Attacks

VLM2Vec: Training Vision-Language Models for Massive Multimodal Embedding Tasks

Progressive Multimodal Reasoning via Active Retrieval

PRAGA: Prototype-aware Graph Adaptive Aggregation for Spatial Multi-modal Omics Analysis

ENCLIP: Ensembling and Clustering-Based Contrastive Language-Image Pretraining for Fashion Multimodal Search with Limited Data and Low-Quality Images

M3DocRAG: Multi-modal Retrieval is What You Need for Multi-page Multi-document Understanding

What Factors Affect Multi-Modal In-Context Learning? An In-Depth Exploration

SANE: Strategic Autonomous Non-Smooth Exploration for Multiple Optima Discovery in Multi-modal and Non-differentiable Black-box Functions

GalleryGPT: Analyzing Paintings with Large Multimodal Models

A Case Study on Visual-Audio-Tactile Cross-Modal Retrieval

In Search of Excellence: SHOA as a Competitive Shrike Optimization Algorithm for Multimodal Problems

Multi-Modal Automatic Prosody Annotation with Contrastive Pretraining of SSWP

RREH: Reconstruction Relations Embedded Hashing for Semi-Paired Cross-Modal Retrieval

MESEN: Exploit Multimodal Data to Design Unimodal Human Activity Recognition with Few Labels

CroSSL: Cross-modal Self-Supervised Learning for Time-series through Latent Masking

Multimodal Neural Databases

CrossBind: Collaborative Cross-Modal Identification of Protein Nucleic-Acid-Binding Residues

Cross-modal Contrastive Learning with Asymmetric Co-attention Network for Video Moment Retrieval

On the Language Encoder of Contrastive Cross-modal Models