多模态大模型 - arXivDaily 专题

2409.11365 2024-10-10 cs.CL 83%

CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration

Jiahui Gao, Renjie Pi, Tianyang Han, Han Wu, Lanqing Hong, Lingpeng Kong, Xin Jiang, Zhenguo Li

专题命中跨模态检索：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL

Comments 10 pages, COLM-2024

2308.14263 2024-10-01 cs.IR cs.MM 83%

Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions

Tianshi Wang, Fengling Li, Lei Zhu, Jingjing Li, Zheng Zhang, Heng Tao Shen

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.MM

2409.04561 2024-09-23 cs.CV 83%

Dual-Level Cross-Modal Contrastive Clustering

Haixin Zhang, Yongjun Li, Dong Huang

专题命中跨模态检索：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments We have found that our paper has many imperfections and incorrect formulas and derivations, and we insist on retracting the manuscript in order to avoid misleading readers.

2409.09282 2024-09-17 cs.LG cs.MM 83%

Turbo your multi-modal classification with contrastive learning

Zhiyu Zhang, Da Liu, Shengqiang Liu, Anna Wang, Jie Gao, Yali Li

专题命中跨模态检索：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.MM

2408.02245 2024-09-17 cs.CV 83%

A Two-Stage Progressive Pre-training using Multi-Modal Contrastive Masked Autoencoders

Muhammad Abdullah Jamal, Omid Mohareri

专题命中跨模态检索：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2404.13400 2024-09-06 cs.CV 83%

HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding

Linhui Xiao, Xiaoshan Yang, Fang Peng, Yaowei Wang, Changsheng Xu

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ACM MM 2024. The project page: https://github.com/linhuixiao/HiVG

2408.14723 2024-08-28 cs.CV cs.IR 83%

Snap and Diagnose: An Advanced Multimodal Retrieval System for Identifying Plant Diseases in the Wild

Tianqi Wei, Zhi Chen, Xin Yu

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2408.08933 2024-08-20 cs.IR cs.AI cs.DB 83%

RoarGraph: A Projected Bipartite Graph for Efficient Cross-Modal Approximate Nearest Neighbor Search

Meng Chen, Kai Zhang, Zhenying He, Yinan Jing, X. Sean Wang

专题命中跨模态检索：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.AI

Comments to be published in PVLDB

2408.05711 2024-08-13 cs.CV 83%

Contrastive masked auto-encoders based self-supervised hashing for 2D image and 3D point cloud cross-modal retrieval

Rukai Wei, Heng Cui, Yu Liu, Yufeng Hou, Yanzhao Xie, Ke Zhou

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by ICME 2024

2407.16307 2024-07-29 cs.MM cs.CR 83%

Multimodal Unlearnable Examples: Protecting Data against Multimodal Contrastive Learning

Xinwei Liu, Xiaojun Jia, Yuan Xun, Siyuan Liang, Xiaochun Cao

专题命中跨模态检索：multimodal(title,abstract)；image-text(abstract)；分类 cs.MM

Comments ACM MM2024

2407.04207 2024-07-24 cs.CV 83%

Elevating All Zero-Shot Sketch-Based Image Retrieval Through Multimodal Prompt Learning

Mainak Singha, Ankit Jha, Divyam Gupta, Pranav Singla, Biplab Banerjee

专题命中跨模态检索：multimodal(title)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted in ECCV 2024

2403.10153 2024-07-16 cs.CV cs.LG 83%

Improving Medical Multi-modal Contrastive Learning with Expert Annotations

Yogesh Kumar, Pekka Marttinen

专题命中跨模态检索：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted to ECCV 2024

2404.00469 2024-07-15 cs.CV 83%

SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs

Yang Miao, Francis Engelmann, Olga Vysotska, Federico Tombari, Marc Pollefeys, Dániel Béla Baráth

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2407.07479 2024-07-11 cs.CV 83%

How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?

Yuxin Chen, Zongyang Ma, Ziqi Zhang, Zhongang Qi, Chunfeng Yuan, Bing Li, Junfu Pu, Ying Shan, Xiaojuan Qi, Weiming Hu

专题命中跨模态检索：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by CVPR 2024

2406.17520 2024-06-26 cs.CV cs.RO 83%

Tell Me Where You Are: Multimodal LLMs Meet Place Recognition

Zonglin Lyu, Juexiao Zhang, Mingxuan Lu, Yiming Li, Chen Feng

专题命中跨模态检索：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

2405.17900 2024-05-29 cs.CL 83%

Enhancing Emotion Recognition in Conversation through Emotional Cross-Modal Fusion and Inter-class Contrastive Learning

Haoxiang Shi, Xulong Zhang, Ning Cheng, Yong Zhang, Jun Yu, Jing Xiao, Jianzong Wang

专题命中跨模态检索：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CL

Comments Accepted by the 20th International Conference on Intelligent Computing (ICIC 2024)

2405.14726 2024-05-24 cs.CV 83%

Distilling Vision-Language Pretraining for Efficient Cross-Modal Retrieval

Young Kyun Jang, Donghyun Kim, Ser-nam Lim

专题命中跨模态检索：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

2405.04741 2024-05-09 cs.CV 83%

All in One Framework for Multimodal Re-identification in the Wild

He Li, Mang Ye, Ming Zhang, Bo Du

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments 12 pages, 3 figure, CVPR 2024

2404.15875 2024-04-25 cs.MM 83%

Simple but Effective Raw-Data Level Multimodal Fusion for Composed Image Retrieval

Haokun Wen, Xuemeng Song, Xiaolin Chen, Yinwei Wei, Liqiang Nie, Tat-Seng Chua

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.MM

Comments ACM SIGIR 2024

2404.01192 2024-04-02 eess.IV cs.CV 83%

iMD4GC: Incomplete Multimodal Data Integration to Advance Precise Treatment Response Prediction and Survival Analysis for Gastric Cancer

Fengtao Zhou, Yingxue Xu, Yanfen Cui, Shenyan Zhang, Yun Zhu, Weiyang He, Jiguang Wang, Xin Wang, Ronald Chan, Louis Ho Shing Lau, Chu Han, Dafu Zhang, Zhenhui Li, Hao Chen

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments 27 pages, 9 figures, 3 tables (under review)

2403.12894 2024-03-21 cs.CV 83%

MEDBind: Unifying Language and Multimodal Medical Data Embeddings

Yuan Gao, Sangwook Kim, David E Austin, Chris McIntosh

专题命中跨模态检索：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

2403.10146 2024-03-18 cs.SD cs.IR eess.AS 83%

Multiscale Matching Driven by Cross-Modal Similarity Consistency for Audio-Text Retrieval

Qian Wang, Jia-Chen Gu, Zhen-Hua Ling

专题命中跨模态检索：cross-modal(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments 5 pages, accepted to ICASSP2024

2312.01714 2024-03-05 cs.CL 83%

Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models

Bingshuai Liu, Chenyang Lyu, Zijun Min, Zhanyu Wang, Jinsong Su, Longyue Wang

专题命中跨模态检索：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments Work in progress

2402.17535 2024-02-28 cs.IR cs.CV 83%

Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control

Thong Nguyen, Mariya Hendriksen, Andrew Yates, Maarten de Rijke

专题命中跨模态检索：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 17 pages, accepted as a full paper at ECIR 2024

2401.11401 2024-01-23 cs.CV 83%

LLMRA: Multi-modal Large Language Model based Restoration Assistant

Xiaoyu Jin, Yuan Shi, Bin Xia, Wenming Yang

专题命中跨模态检索：multi-modal(title,abstract)；MLLM(abstract)；分类 cs.CV

2401.08206 2024-01-17 cs.IR cs.CL 83%

Generative Multi-Modal Knowledge Retrieval with Large Language Models

Xinwei Long, Jiali Zeng, Fandong Meng, Zhiyuan Ma, Kaiyan Zhang, Bowen Zhou, Jie Zhou

专题命中跨模态检索：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments Accepted to AAAI 2024

2401.05736 2024-01-12 cs.CL cs.IR 83%

Cross-modal Retrieval for Knowledge-based Visual Question Answering

Paul Lerner, Olivier Ferret, Camille Guinaudeau

专题命中跨模态检索：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL

2401.00849 2024-01-02 cs.CV 83%

COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training

Alex Jinpeng Wang, Linjie Li, Kevin Qinghong Lin, Jianfeng Wang, Kevin Lin, Zhengyuan Yang, Lijuan Wang, Mike Zheng Shou

专题命中跨模态检索：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments 16 pages; Website: http://fingerrec.github.io/cosmo

2312.12334 2023-12-20 cs.CL 83%

PowMix: A Versatile Regularizer for Multimodal Sentiment Analysis

Efthymios Georgiou, Yannis Avrithis, Alexandros Potamianos

专题命中跨模态检索：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments Preprint

2312.03796 2023-12-08 cs.LG cs.AI 83%

Multi-Scale and Multi-Modal Contrastive Learning Network for Biomedical Time Series

Hongbo Guo, Xinzi Xu, Hao Wu, Guoxing Wang

专题命中跨模态检索：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments 4 pages, 3 figures, submitted to ICASSP 2024

1. 跨模态检索 3437 篇

CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibration

Cross-Modal Retrieval: A Systematic Review of Methods and Future Directions

Dual-Level Cross-Modal Contrastive Clustering

Turbo your multi-modal classification with contrastive learning

A Two-Stage Progressive Pre-training using Multi-Modal Contrastive Masked Autoencoders

HiVG: Hierarchical Multimodal Fine-grained Modulation for Visual Grounding

Snap and Diagnose: An Advanced Multimodal Retrieval System for Identifying Plant Diseases in the Wild

RoarGraph: A Projected Bipartite Graph for Efficient Cross-Modal Approximate Nearest Neighbor Search

Contrastive masked auto-encoders based self-supervised hashing for 2D image and 3D point cloud cross-modal retrieval

Multimodal Unlearnable Examples: Protecting Data against Multimodal Contrastive Learning

Elevating All Zero-Shot Sketch-Based Image Retrieval Through Multimodal Prompt Learning

Improving Medical Multi-modal Contrastive Learning with Expert Annotations

SceneGraphLoc: Cross-Modal Coarse Visual Localization on 3D Scene Graphs

How to Make Cross Encoder a Good Teacher for Efficient Image-Text Retrieval?

Tell Me Where You Are: Multimodal LLMs Meet Place Recognition

Enhancing Emotion Recognition in Conversation through Emotional Cross-Modal Fusion and Inter-class Contrastive Learning

Distilling Vision-Language Pretraining for Efficient Cross-Modal Retrieval

All in One Framework for Multimodal Re-identification in the Wild

Simple but Effective Raw-Data Level Multimodal Fusion for Composed Image Retrieval

iMD4GC: Incomplete Multimodal Data Integration to Advance Precise Treatment Response Prediction and Survival Analysis for Gastric Cancer

MEDBind: Unifying Language and Multimodal Medical Data Embeddings

Multiscale Matching Driven by Cross-Modal Similarity Consistency for Audio-Text Retrieval

Retrieval-augmented Multi-modal Chain-of-Thoughts Reasoning for Large Language Models

Multimodal Learned Sparse Retrieval with Probabilistic Expansion Control

LLMRA: Multi-modal Large Language Model based Restoration Assistant

Generative Multi-Modal Knowledge Retrieval with Large Language Models

Cross-modal Retrieval for Knowledge-based Visual Question Answering

COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training

PowMix: A Versatile Regularizer for Multimodal Sentiment Analysis

Multi-Scale and Multi-Modal Contrastive Learning Network for Biomedical Time Series