多模态大模型

SimMMDG: A Simple and Effective Framework for Multi-modal Domain Generalization

Hao Dong, Ismail Nejjar, Han Sun, Eleni Chatzi, Olga Fink

专题命中 跨模态检索 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2023

A scoping review on multimodal deep learning in biomedical images and texts

Zhaoyi Sun, Mingquan Lin, Qingqing Zhu, Qianqian Xie, Fei Wang, Zhiyong Lu, Yifan Peng

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments This paper has been accepted by the Journal of Biomedical Informatics

Journal ref Journal of Biomedical Informatics, Volume 146, October 2023, 104482

Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition in Conversation

Shihao Zou, Xianying Huang, Xudong Shen

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted to ACM MM 2023

CWCL: Cross-Modal Transfer with Continuously Weighted Contrastive Loss

Rakshith Sharma Srinivasa, Jaejin Cho, Chouchang Yang, Yashas Malur Saidutta, Ching-Hua Lee, Yilin Shen, Hongxia Jin

专题命中 跨模态检索 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted to Neural Information Processing Systems (NeurIPS) 2023 conference

Dual-view Curricular Optimal Transport for Cross-lingual Cross-modal Retrieval

Yabing Wang, Shuhui Wang, Hao Luo, Jianfeng Dong, Fan Wang, Meng Han, Xun Wang, Meng Wang

专题命中 跨模态检索 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Towards Fast and Accurate Image-Text Retrieval with Self-Supervised Fine-Grained Alignment

Jiamin Zhuang, Jing Yu, Yang Ding, Xiangyan Qu, Yue Hu

专题命中 跨模态检索 ：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted in IEEE Transactions on Multimedia (TMM)

Journal ref IEEE Transactions on Multimedia ( Early Access ), 29 May 2023

CleanCLIP: Mitigating Data Poisoning Attacks in Multimodal Contrastive Learning

Hritik Bansal, Nishad Singhi, Yu Yang, Fan Yin, Aditya Grover, Kai-Wei Chang

专题命中 跨模态检索 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments 22 pages. Accepted at ICCV 2023

Tackling Ambiguity with Images: Improved Multimodal Machine Translation and Contrastive Evaluation

Matthieu Futeral, Cordelia Schmid, Ivan Laptev, Benoît Sagot, Rachel Bawden

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted to ACL 2023

UniS-MMC: Multimodal Classification via Unimodality-supervised Multimodal Contrastive Learning

Heqing Zou, Meng Shen, Chen Chen, Yuchen Hu, Deepu Rajan, Eng Siong Chng

专题命中 跨模态检索 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments ACL 2023 Findings

REVEAL: Retrieval-Augmented Visual-Language Pre-Training with Multi-Source Multimodal Knowledge Memory

Ziniu Hu, Ahmet Iscen, Chen Sun, Zirui Wang, Kai-Wei Chang, Yizhou Sun, Cordelia Schmid, David A. Ross, Alireza Fathi

专题命中 跨模态检索 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Published on CVPR 2023

RAMM: Retrieval-augmented Biomedical Visual Question Answering with Multi-modal Pre-training

Zheng Yuan, Qiao Jin, Chuanqi Tan, Zhengyun Zhao, Hongyi Yuan, Fei Huang, Songfang Huang

专题命中 跨模态检索 ：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

DUET: Cross-modal Semantic Grounding for Contrastive Zero-shot Learning

Zhuo Chen, Yufeng Huang, Jiaoyan Chen, Yuxia Geng, Wen Zhang, Yin Fang, Jeff Z. Pan, Huajun Chen

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments AAAI 2023 (Oral). Repository: https://github.com/zjukg/DUET

Multi-scale multi-modal micro-expression recognition algorithm based on transformer

Fengping Wang, Jie Li, Chun Qi, Lin Wang, Pan Wang

专题命中 跨模态检索 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Scale-Semantic Joint Decoupling Network for Image-text Retrieval in Remote Sensing

Chengyu Zheng, Ning song, Ruoyu Zhang, Lei Huang, Zhiqiang Wei, Jie Nie

专题命中 跨模态检索 ：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Look Before You Leap: Improving Text-based Person Retrieval by Learning A Consistent Cross-modal Common Manifold

Zijie Wang, Aichun Zhu, Jingyi Xue, Xili Wan, Chao Liu, Tian Wang, Yifeng Li

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted on ACM MM '22. arXiv admin note: text overlap with arXiv:2209.05773

Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning

Yabing Wang, Jianfeng Dong, Tianxiang Liang, Minsong Zhang, Rui Cai, Xun Wang

专题命中 跨模态检索 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Comments Accepted by ACM MM 2022. Code and data are available at https://github.com/HuiGuanLab/nrccr

Language Guided Networks for Cross-modal Moment Retrieval

Kun Liu, Huadong Ma, Chuang Gan

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Intra-Modal Constraint Loss For Image-Text Retrieval

Jianan Chen, Lu Zhang, Qiong Wang, Cong Bai, Kidiyo Kpalma

专题命中 跨模态检索 ：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

BOSS: Bottom-up Cross-modal Semantic Composition with Hybrid Counterfactual Training for Robust Content-based Image Retrieval

Wenqiao Zhang, Jiannan Guo, Mengze Li, Haochen Shi, Shengyu Zhang, Juncheng Li, Siliang Tang, Yueting Zhuang

专题命中 跨模态检索 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Contrastive Cross-Modal Knowledge Sharing Pre-training for Vision-Language Representation Learning and Retrieval

Keyu Wen, Zhenshan Tan, Qingrong Cheng, Cheng Chen, Xiaodong Gu

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Entity-Graph Enhanced Cross-Modal Pretraining for Instance-level Product Retrieval

Xiao Dong, Xunlin Zhan, Yunchao Wei, Xiaoyong Wei, Yaowei Wang, Minlong Lu, Xiaochun Cao, Xiaodan Liang

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM

COTS: Collaborative Two-Stream Vision-Language Pre-Training Model for Cross-Modal Retrieval

Haoyu Lu, Nanyi Fei, Yuqi Huo, Yizhao Gao, Zhiwu Lu, Ji-Rong Wen

专题命中 跨模态检索 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted by CVPR2022

Uncertainty-based Cross-Modal Retrieval with Probabilistic Representations

Leila Pishdad, Ran Zhang, Konstantinos G. Derpanis, Allan Jepson, Afsaneh Fazly

专题命中 跨模态检索 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 13 pages, 7 figures

MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering

Yang Ding, Jing Yu, Bang Liu, Yue Hu, Mingxin Cui, Qi Wu

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by CVPR2022

Retrieve Fast, Rerank Smart: Cooperative and Joint Approaches for Improved Cross-Modal Retrieval

Gregor Geigle, Jonas Pfeiffer, Nils Reimers, Ivan Vulić, Iryna Gurevych

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments TACL 2022

Discriminative Supervised Subspace Learning for Cross-modal Retrieval

Haoming Zhang, Xiao-Jun Wu, Tianyang Xu, Donglin Zhang

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Constructing Phrase-level Semantic Labels to Form Multi-Grained Supervision for Image-Text Retrieval

Zhihao Fan, Zhongyu Wei, Zejun Li, Siyuan Wang, Haijun Shan, Xuanjing Huang, Jianqing Fan

专题命中 跨模态检索 ：image-text(title)；multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Multimodal Metric Learning for Tag-based Music Retrieval

Minz Won, Sergio Oramas, Oriol Nieto, Fabien Gouyon, Xavier Serra

专题命中 跨模态检索 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

Comments 5 pages, 2 figures, submitted to ICASSP 2021

DIME: An Online Tool for the Visual Comparison of Cross-Modal Retrieval Models

Tony Zhao, Jaeyoung Choi, Gerald Friedland

专题命中 跨模态检索 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM

Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos

Zhu Zhang, Zhijie Lin, Zhou Zhao, Zhenxin Xiao

专题命中 跨模态检索 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by SIGIR 2019 as a full paper

Journal ref SIGIR, 2019, pages 655-664