多模态大模型

Robust Semi-supervised Multimodal Medical Image Segmentation via Cross Modality Collaboration

Xiaogen Zhou, Yiyou Sun, Min Deng, Winnie Chiu Wing Chu, Qi Dou

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data

Spencer Whitehead, Jacob Phillips, Sean Hendryx

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

KAN-RCBEVDepth: A multi-modal fusion algorithm in object detection for autonomous driving

Zhihao Lai, Chuanhao Liu, Shihui Sheng, Zhiqiang Zhang

专题命中 多模态训练与对齐 ：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.AI

Multimodal Ensemble with Conditional Feature Fusion for Dysgraphia Diagnosis in Children from Handwriting Samples

Jayakanth Kunhoth, Somaya Al-Maadeed, Moutaz Saleh, Younes Akbari

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Self-Supervised Multimodal Learning: A Survey

Yongshuo Zong, Oisin Mac Aodha, Timothy Hospedales

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted to IEEE T-PAMI

MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examples

Tao Chen, Enwei Zhang, Yuting Gao, Ke Li, Xing Sun, Yan Zhang, Hui Li, Rongrong Ji

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments TOMM 2024

MU-MAE: Multimodal Masked Autoencoders-Based One-Shot Learning

Rex Liu, Xin Liu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.MM

Comments IEEE MIPR 2024

LoginMEA: Local-to-Global Interaction Network for Multi-modal Entity Alignment

Taoyu Su, Xinghua Zhang, Jiawei Sheng, Zhenyu Zhang, Tingwen Liu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL、cs.MM

Comments Accepted by ECAI 2024

LLAVADI: What Matters For Multimodal Large Language Models Distillation

Shilin Xu, Xiangtai Li, Haobo Yuan, Lu Qi, Yunhai Tong, Ming-Hsuan Yang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

IBMEA: Exploring Variational Information Bottleneck for Multi-modal Entity Alignment

Taoyu Su, Jiawei Sheng, Shicheng Wang, Xinghua Zhang, Hongbo Xu, Tingwen Liu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CL、cs.MM

Comments Accepted by ACM MM 2024

MMEarth: Exploring Multi-Modal Pretext Tasks For Geospatial Representation Learning

Vishal Nedungadi, Ankit Kariryaa, Stefan Oehmcke, Serge Belongie, Christian Igel, Nico Lang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted for ECCV 2024. Data and code: https://vishalned.github.io/mmearth Update arXiv v2 (ECCV): 1. Dataset fix: Removed duplicates and corrected ERA5 yearly statistics. 2. Data augmentation fix: Random crops are now aligned. 3. Test metrics fix: Metrics are now overall instead of mini-batch averages, matching GEO-Bench metrics. 4. Pretrained on MMEarth v001 & evaluated on GEO-Bench v1.0

Exploring Fusion Techniques in Multimodal AI-Based Recruitment: Insights from FairCVdb

Swati Swati, Arjun Roy, Eirini Ntoutsi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

FoRA: Low-Rank Adaptation Model beyond Multimodal Siamese Network

Weiying Xie, Yusi Zhang, Tianlin Hui, Jiaqing Zhang, Jie Lei, Yunsong Li

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Pay Less On Clinical Images: Asymmetric Multi-Modal Fusion Method For Efficient Multi-Label Skin Lesion Classification

Peng Tang, Tobias Lasser

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

Robust Multimodal Learning via Representation Decoupling

Shicai Wei, Yang Luo, Yuji Wang, Chunbo Luo

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments ECCV2024 17 pages

Understanding Alignment in Multimodal LLMs: A Comprehensive Study

Elmira Amirloo, Jean-Philippe Fauconnier, Christoph Roesmann, Christian Kerl, Rinu Boney, Yusu Qian, Zirui Wang, Afshin Dehghan, Yinfei Yang, Zhe Gan, Peter Grasch

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Similarity Guided Multimodal Fusion Transformer for Semantic Location Prediction in Social Media

Zhizhen Zhang, Ning Wang, Haojie Li, Zhihui Wang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

BEACON: Balancing Convenience and Nutrition in Meals With Long-Term Group Recommendations and Reasoning on Multimodal Recipes

Vansh Nagpal, Siva Likitha Valluru, Kausik Lakkaraju, Biplav Srivastava

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 6 pages (including references), 1 figure, 2 tables

Xmodel-VLM: A Simple Baseline for Multimodal Vision Language Model

Wanting Xu, Yang Liu, Langping He, Xucheng Huang, Ling Jiang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Multimodal Large Language Models with Fusion Low Rank Adaptation for Device Directed Speech Detection

Shruti Palaskar, Oggi Rudovic, Sameer Dharur, Florian Pesce, Gautam Krishna, Aswin Sivaraman, Jack Berkowitz, Ahmed Hussen Abdelaziz, Saurabh Adya, Ahmed Tewfik

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted at Interspeech 2024

Towards Robust Instruction Tuning on Multimodal Large Language Models

Wei Han, Hui Chen, Soujanya Poria

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 24 pages, 7 figures

Multimodal Belief Prediction

John Murzaku, Adil Soubki, Owen Rambow

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments John Murzaku and Adil Soubki contributed equally to this work

Journal ref Interspeech 2024

Mixture of Rationale: Multi-Modal Reasoning Mixture for Visual Question Answering

Tao Li, Linjun Shou, Xuejun Liu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI

II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering

Jihyung Kil, Farideh Tavazoee, Dongyeop Kang, Joo-Kyung Kim

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted to ACL 2024 Findings

Efficient Ensemble for Multimodal Punctuation Restoration using Time-Delay Neural Network

Xing Yi Liu, Homayoon Beigi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments 6 pages, 1 figure, 5 tables, paper at IMCOM 2024, technical report at Recognition Technologies, Inc

Journal ref 2024 18th International Conference on Ubiquitous Information Management and Communication (IMCOM), 2024, pp. 1-6

Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models

Yue Zhang, Hehe Fan, Yi Yang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Distilling Privileged Multimodal Information for Expression Recognition using Optimal Transport

Muhammad Haseeb Aslam, Muhammad Osama Zeeshan, Soufiane Belharbi, Marco Pedersoli, Alessandro Koerich, Simon Bacon, Eric Granger

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

A review of deep learning-based information fusion techniques for multimodal medical image classification

Yihao Li, Mostafa El Habib Daho, Pierre-Henri Conze, Rachid Zeghlache, Hugo Le Boité, Ramin Tadayoni, Béatrice Cochener, Mathieu Lamard, Gwenolé Quellec

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Quantifying and Enhancing Multi-modal Robustness with Modality Preference

Zequn Yang, Yake Wei, Ce Liang, Di Hu

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV、cs.MM

Comments Accepted to ICLR 2024

Exploring the Transferability of Visual Prompting for Multimodal Large Language Models

Yichi Zhang, Yinpeng Dong, Siyuan Zhang, Tianzan Min, Hang Su, Jun Zhu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted in CVPR 2024 as Poster (Highlight)