多模态大模型

Retrieval Meets Reasoning: Even High-school Textbook Knowledge Benefits Multimodal Reasoning

Cheng Tan, Jingxuan Wei, Linzhuang Sun, Zhangyang Gao, Siyuan Li, Bihui Yu, Ruifeng Guo, Stan Z. Li

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV

Comments Under review

Reverse Image Retrieval Cues Parametric Memory in Multimodal LLMs

Jialiang Xu, Michael Moor, Jure Leskovec

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL

${M^2D}$NeRF: Multi-Modal Decomposition NeRF with 3D Feature Fields

Ning Wang, Lefei Zhang, Angel X Chang

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CV

COM3D: Leveraging Cross-View Correspondence and Cross-Modal Mining for 3D Retrieval

Hao Wu, Ruochong LI, Hao Wang, Hui Xiong

专题命中 跨模态检索 ：cross-modal(title,abstract)；分类 cs.CV

Comments Accepted by ICME 2024 oral

Revisiting Multimodal Emotion Recognition in Conversation from the Perspective of Graph Spectrum

Tao Meng, Fuchen Zhang, Yuntao Shou, Wei Ai, Nan Yin, Keqin Li

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL

Comments 10 pages, 4 figures

Next Visit Diagnosis Prediction via Medical Code-Centric Multimodal Contrastive EHR Modelling with Hierarchical Regularisation

Heejoon Koo

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.AI

Comments Accepted to EACL 2024 (The 18th Conference of the European Chapter of the Association for Computational Linguistics)

Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMs

Shengbang Tong, Zhuang Liu, Yuexiang Zhai, Yi Ma, Yann LeCun, Saining Xie

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV

Comments Project page: https://tsb0601.github.io/mmvp_blog/

Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understanding

Le Zhang, Rabiul Awal, Aishwarya Agrawal

专题命中 跨模态检索 ：cross-modal(title)；image-text(abstract)；分类 cs.CV

Comments CVPR 2024

Large Language Models and Multimodal Retrieval for Visual Word Sense Disambiguation

Anastasia Kritharoula, Maria Lymperaiou, Giorgos Stamou

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL

Comments Conference on Empirical Methods in Natural Language Processing (EMNLP) 2023

Journal ref Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

Global Contrastive Training for Multimodal Electronic Health Records with Language Supervision

Yingbo Ma, Suraj Kolla, Zhenhong Hu, Dhruv Kaliraman, Victoria Nolan, Ziyuan Guan, Yuanfang Ren, Brooke Armfield, Tezcan Ozrazgat-Baslanti, Jeremy A. Balch, Tyler J. Loftus, Parisa Rashidi, Azra Bihorac, Benjamin Shickel

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL

Comments 12 pages, 3 figures. arXiv admin note: text overlap with arXiv:2403.04012

MMIDR: Teaching Large Language Model to Interpret Multimodal Misinformation via Knowledge Distillation

Longzheng Wang, Xiaohan Xu, Lei Zhang, Jiarui Lu, Yongxiu Xu, Hongbo Xu, Minghao Tang, Chuang Zhang

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL

Comments 10 pages, 3 figures

KDMCSE: Knowledge Distillation Multimodal Sentence Embeddings with Adaptive Angular margin Contrastive Learning

Cong-Duy Nguyen, Thong Nguyen, Xiaobao Wu, Anh Tuan Luu

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL

Comments Accepted to NAACL 2024

CMViM: Contrastive Masked Vim Autoencoder for 3D Multi-modal Representation Learning for AD classification

Guangqian Yang, Kangrui Du, Zhihan Yang, Ye Du, Yongping Zheng, Shujun Wang

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CV

Comments 11 pages, 1 figure

Path-GPTOmic: A Balanced Multi-modal Learning Framework for Survival Outcome Prediction

Hongxiao Wang, Yang Yang, Zhuo Zhao, Pengfei Gu, Nishchal Sapkota, Danny Z. Chen

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted by IEEE International Symposium on Biomedical Imaging (ISBI 2024)

SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM

Jielin Qiu, Andrea Madotto, Zhaojiang Lin, Paul A. Crook, Yifan Ethan Xu, Xin Luna Dong, Christos Faloutsos, Lei Li, Babak Damavandi, Seungwhan Moon

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV

A Privacy-Preserving Framework with Multi-Modal Data for Cross-Domain Recommendation

Li Wang, Lei Sang, Quangui Zhang, Qiang Wu, Min Xu

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.AI

BiVRec: Bidirectional View-based Multimodal Sequential Recommendation

Jiaxi Hu, Jingtong Gao, Xiangyu Zhao, Yuehong Hu, Yuxuan Liang, Yiqi Wang, Ming He, Zitao Liu, Hongzhi Yin

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.AI

BadCLIP: Dual-Embedding Guided Backdoor Attack on Multimodal Contrastive Learning

Siyuan Liang, Mingli Zhu, Aishan Liu, Baoyuan Wu, Xiaochun Cao, Ee-Chien Chang

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV

Comments The paper lacks some work that needs to be cited

Journal ref CVPR 2024

Robots Autonomously Detecting People: A Multimodal Deep Contrastive Learning Method Robust to Intraclass Variations

Angus Fung, Beno Benhabib, Goldie Nejat

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV

Multimodal Learned Sparse Retrieval for Image Suggestion

Thong Nguyen, Mariya Hendriksen, Andrew Yates

专题命中 跨模态检索 ：multimodal(title)；multi-modal(abstract)；分类 cs.MM

Comments 5 pages, TREC 2023

REALM: RAG-Driven Enhancement of Multimodal Electronic Health Records Analysis via Large Language Models

Yinghao Zhu, Changyu Ren, Shiyun Xie, Shukai Liu, Hangyuan Ji, Zixiang Wang, Tao Sun, Long He, Zhoujun Li, Xi Zhu, Chengwei Pan

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.AI

Transformer-empowered Multi-modal Item Embedding for Enhanced Image Search in E-Commerce

Chang Liu, Peng Hou, Anxiang Zeng, Han Yu

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted by IAAI 2024

Multi-modal Molecule Structure-text Model for Text-based Retrieval and Editing

Shengchao Liu, Weili Nie, Chengpeng Wang, Jiarui Lu, Zhuoran Qiao, Ling Liu, Jian Tang, Chaowei Xiao, Anima Anandkumar

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CL

Candidate Set Re-ranking for Composed Image Retrieval with Dual Multi-modal Encoder

Zheyuan Liu, Weixuan Sun, Damien Teney, Stephen Gould

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted at TMLR, 19 pages, 8 figures

Multimodal Data Curation via Object Detection and Filter Ensembles

Tzu-Heng Huang, Changho Shin, Sui Jiet Tay, Dyah Adila, Frederic Sala

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV

Comments Appeared in the Workshop of Towards the Next Generation of Computer Vision Datasets (TNGCV) on ICCV 2023

SignVTCL: Multi-Modal Continuous Sign Language Recognition Enhanced by Visual-Textual Contrastive Learning

Hao Chen, Jiaze Wang, Ziyu Guo, Jinpeng Li, Donghao Zhou, Bian Wu, Chenyong Guan, Guangyong Chen, Pheng-Ann Heng

专题命中 跨模态检索 ：multi-modal(title,abstract)；分类 cs.CV

MATE-Pred: Multimodal Attention-based TCR-Epitope interaction Predictor

Etienne Goffinet, Raghvendra Mall, Ankita Singh, Rahul Kaushik, Filippo Castiglione

专题命中 跨模态检索 ：multimodal(title)；multi-modal(abstract)；分类 cs.AI

Comments Patent pending: U.S. Provisional Application No. 63/603,952

Prototype-based Aleatoric Uncertainty Quantification for Cross-modal Retrieval

Hao Li, Jingkuan Song, Lianli Gao, Xiaosu Zhu, Heng Tao Shen

专题命中 跨模态检索 ：cross-modal(title,abstract)；分类 cs.CV

Comments Accepted to NeurIPS 2023

CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs

Daoan Zhang, Junming Yang, Hanjia Lyu, Zijian Jin, Yuan Yao, Mingkai Chen, Jiebo Luo

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CV

A Two-Stage Multimodal Emotion Recognition Model Based on Graph Contrastive Learning

Wei Ai, FuChen Zhang, Tao Meng, YunTao Shou, HongEn Shao, Keqin Li

专题命中 跨模态检索 ：multimodal(title,abstract)；分类 cs.CL

Comments 9 pages, 3 figures