多模态大模型

LLaMP: Large Language Model Made Powerful for High-fidelity Materials Knowledge Retrieval and Distillation

Yuan Chiang, Elvis Hsieh, Chia-Hong Chou, Janosh Riebesell

专题命中 跨模态检索 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 32 pages, 5 figures

Temporal Image Caption Retrieval Competition -- Description and Results

Jakub Pokrywka, Piotr Wierzchoń, Kornel Weryszko, Krzysztof Jassem

专题命中 跨模态检索 ：multimodal(abstract)；分类 cs.CV、cs.CL

Journal ref Proceedings of the 18th Conference on Computer Science and Intelligence Systems, M. Ganzha, L. Maciaszek, M. Paprzycki, D. Ślęzak (eds). ACSIS, Vol. 35, pages 1331-1336 (2023)

Coarse-to-fine Alignment Makes Better Speech-image Retrieval

Lifeng Zhou, Yuke Li

专题命中 跨模态检索 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Limitations in Employing Natural Language Supervision for Sensor-Based Human Activity Recognition -- And Ways to Overcome Them

Harish Haresamudram, Apoorva Beedu, Mashfiqui Rabbi, Sankalita Saha, Irfan Essa, Thomas Ploetz

专题命中 跨模态检索 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Context-aware Visual Storytelling with Visual Prefix Tuning and Contrastive Learning

Yingjin Song, Denis Paperno, Albert Gatt

专题命中 跨模态检索 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments 18 pages, 12 figures, accepted by INLG 2024

Improving Composed Image Retrieval via Contrastive Learning with Scaling Positives and Negatives

Zhangchi Feng, Richong Zhang, Zhijie Nie

专题命中 跨模态检索 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted to ACM MM 2024 Regular Papers

X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs

Sirnam Swetha, Jinyu Yang, Tal Neiman, Mamshad Nayeem Rizve, Son Tran, Benjamin Yao, Trishul Chilimbi, Mubarak Shah

专题命中 跨模态检索 ：multimodal(abstract)；分类 cs.CV、cs.MM

Comments Accepted at ECCV2024

Reproducible scaling laws for contrastive language-image learning

Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, Jenia Jitsev

专题命中 跨模态检索 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2023. Version with minor extension. Original: CVPR2023/html/Cherti_Reproducible_Scaling_Laws_for_Contrastive_Language-Image_Learning_CVPR_2023_paper" target="_blank" rel="noopener">https://openaccess.thecvf.com/content/CVPR2023/html/Cherti_Reproducible_Scaling_Laws_for_Contrastive_Language-Image_Learning_CVPR_2023_paper

Journal ref Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 2818-2829

CEIA: CLIP-Based Event-Image Alignment for Open-World Event-Based Understanding

Wenhao Xu, Wenming Weng, Yueyi Zhang, Zhiwei Xiong

专题命中 跨模态检索 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Semantic Compositions Enhance Vision-Language Contrastive Learning

Maxwell Aladago, Lorenzo Torresani, Soroush Vosoughi

专题命中 跨模态检索 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Nomic Embed Vision: Expanding the Latent Space

Zach Nussbaum, Brandon Duderstadt, Andriy Mulyar

专题命中 跨模态检索 ：multimodal(abstract)；分类 cs.CV、cs.AI

BEV-TSR: Text-Scene Retrieval in BEV Space for Autonomous Driving

Tao Tang, Dafeng Wei, Zhengyu Jia, Tian Gao, Changwei Cai, Chengkai Hou, Peng Jia, Kun Zhan, Haiyang Sun, Jingchen Fan, Yixing Zhao, Fu Liu, Xiaodan Liang, Xianpeng Lang, Yang Wang

专题命中 跨模态检索 ：cross-modal(abstract)；分类 cs.CV、cs.AI

It's Not a Modality Gap: Characterizing and Addressing the Contrastive Gap

Abrar Fahim, Alex Murphy, Alona Fyshe

专题命中 跨模态检索 ：multi-modal(abstract)；分类 cs.CV、cs.CL

CSS: Contrastive Semantic Similarity for Uncertainty Quantification of LLMs

Shuang Ao, Stefan Rueger, Advaith Siddharthan

专题命中 跨模态检索 ：image-text(abstract)；分类 cs.CL、cs.AI

Comments The paper is accepted by The Conference on Uncertainty in Artificial Intelligence (UAI), 2024

Retrieval Augmented Structured Generation: Business Document Information Extraction As Tool Use

Franz Louis Cesista, Rui Aguiar, Jason Kim, Paolo Acilo

专题命中 跨模态检索 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Accepted by IEEE 7th International Conference on Multimedia Information Processing and Retrieval (MIPR), 2024

Invisible Relevance Bias: Text-Image Retrieval Models Prefer AI-Generated Images

Shicheng Xu, Danyang Hou, Liang Pang, Jingcheng Deng, Jun Xu, Huawei Shen, Xueqi Cheng

专题命中 跨模态检索 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by SIGIR 2024

MVBIND: Self-Supervised Music Recommendation For Videos Via Embedding Space Binding

Jiajie Teng, Huiyu Duan, Yucheng Zhu, Sijing Wu, Guangtao Zhai

专题命中 跨模态检索 ：cross-modal(abstract)；分类 cs.CV、cs.MM

Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models

Yifei Ming, Yixuan Li

专题命中 跨模态检索 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments The paper is accepted at ICML 2024

Large Language Model Informed Patent Image Retrieval

Hao-Cheng Lo, Jung-Mei Chu, Jieh Hsiang, Chun-Chieh Cho

专题命中 跨模态检索 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments 8 pages. Under review

Fooling Contrastive Language-Image Pre-trained Models with CLIPMasterPrints

Matthias Freiberger, Peter Kun, Christian Igel, Anders Sundnes Løvlie, Sebastian Risi

专题命中 跨模态检索 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments This work was supported by a research grant (40575) from VILLUM FONDEN

Vision-and-Language Navigation via Causal Learning

Liuyi Wang, Zongtao He, Ronghao Dang, Mengjiao Shen, Chengju Liu, Qijun Chen

专题命中 跨模态检索 ：cross-modal(abstract)；分类 cs.CV、cs.AI

WILBUR: Adaptive In-Context Learning for Robust and Accurate Web Agents

Michael Lutz, Arth Bohra, Manvel Saroyan, Artem Harutyunyan, Giovanni Campagna

专题命中 跨模态检索 ：multi-modal(abstract)；分类 cs.CL、cs.AI

PetKaz at SemEval-2024 Task 3: Advancing Emotion Classification with an LLM for Emotion-Cause Pair Extraction in Conversations

Roman Kazakov, Kseniia Petukhova, Ekaterina Kochmar

专题命中 跨模态检索 ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 8 pages, 7 figures, 2 tables, to be published in the Proceedings of the 18th International Workshop on Semantic Evaluation (SemEval-2024), for associated code, see https://github.com/sachertort/petkaz-semeval-ecac

Sparse Concept Bottleneck Models: Gumbel Tricks in Contrastive Learning

Andrei Semenov, Vladimir Ivanov, Aleksandr Beznosikov, Alexander Gasnikov

专题命中 跨模态检索 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments 23 pages, 1 algorithm, 36 figures

CFIR: Fast and Effective Long-Text To Image Retrieval for Large Corpora

Zijun Long, Xuri Ge, Richard Mccreadie, Joemon Jose

专题命中 跨模态检索 ：multimodal(abstract)；分类 cs.CV、cs.AI

Lexicon-Level Contrastive Visual-Grounding Improves Language Modeling

Chengxu Zhuang, Evelina Fedorenko, Jacob Andreas

专题命中 跨模态检索 ：multimodal(abstract)；分类 cs.CL、cs.AI

MOFI: Learning Image Representations from Noisy Entity Annotated Images

Wentao Wu, Aleksei Timofeev, Chen Chen, Bowen Zhang, Kun Duan, Shuangning Liu, Yantao Zheng, Jonathon Shlens, Xianzhi Du, Zhe Gan, Yinfei Yang

专题命中 跨模态检索 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted to ICLR 2024

Tri-Modal Motion Retrieval by Learning a Joint Embedding Space

Kangning Yin, Shihao Zou, Yuxuan Ge, Zheng Tian

专题命中 跨模态检索 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Contrastive Learning Is Spectral Clustering On Similarity Graph

Zhiquan Tan, Yifan Zhang, Jingqin Yang, Yang Yuan

专题命中 跨模态检索 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments ICLR 2024; We express our gratitude to the anonymous reviewers for their valuable feedback

AM^2-EmoJE: Adaptive Missing-Modality Emotion Recognition in Conversation via Joint Embedding Learning

Naresh Kumar Devulapally, Sidharth Anand, Sreyasee Das Bhattacharjee, Junsong Yuan

专题命中 跨模态检索 ：multimodal(abstract)；分类 cs.CV、cs.AI