多模态大模型

Multimodal Cross-Task Interaction for Survival Analysis in Whole Slide Pathological Images

Songhan Jiang, Zhengyu Gan, Linghan Cai, Yifeng Wang, Yongbing Zhang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Multi-Modal Vision Transformers for Crop Mapping from Satellite Image Time Series

Theresa Follath, David Mickisch, Jan Hemmerling, Stefan Erasmi, Marcel Schwieder, Begüm Demir

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Comments 5 pages, 2 figures, 1 table. Accepted at IEEE International Geoscience and Remote Sensing Symposium (IGARSS) 2024. Our code is available at https://git.tu-berlin.de/rsim/mmtsvit

DV-3DLane: End-to-end Multi-modal 3D Lane Detection with Dual-view Representation

Yueru Luo, Shuguang Cui, Zhen Li

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted at ICLR2024

Multimodal Deformable Image Registration for Long-COVID Analysis Based on Progressive Alignment and Multi-perspective Loss

Jiahua Li, James T. Grist, Fergus V. Gleeson, Bartłomiej W. Papież

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Converging Dimensions: Information Extraction and Summarization through Multisource, Multimodal, and Multilingual Fusion

Pranav Janjani, Mayank Palan, Sarvesh Shirude, Ninad Shegokar, Sunny Kumar, Faruk Kazi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Comments 11 pages, 3 figures

M2CURL: Sample-Efficient Multimodal Reinforcement Learning via Self-Supervised Representation Learning for Robotic Manipulation

Fotios Lygerakis, Vedant Dave, Elmar Rueckert

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Comments Project website: https://sites.google.com/view/M2CURL/home

FlexCare: Leveraging Cross-Task Synergy for Flexible Multimodal Healthcare Prediction

Muhao Xu, Zhenfeng Zhu, Youru Li, Shuai Zheng, Yawei Zhao, Kunlun He, Yao Zhao

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Comments Accepted by KDD 2024 (Research Track)

Tuning Large Multimodal Models for Videos using Reinforcement Learning from AI Feedback

Daechul Ahn, Yura Choi, Youngjae Yu, Dongyeop Kang, Jonghyun Choi

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Comments ACL 2024

AstroCLIP: A Cross-Modal Foundation Model for Galaxies

Liam Parker, Francois Lanusse, Siavash Golkar, Leopoldo Sarra, Miles Cranmer, Alberto Bietti, Michael Eickenberg, Geraud Krawezik, Michael McCabe, Ruben Ohana, Mariel Pettee, Bruno Regaldo-Saint Blancard, Tiberiu Tesileanu, Kyunghyun Cho, Shirley Ho

专题命中 多模态训练与对齐 ：cross-modal(title,abstract)；分类 cs.AI

Comments 18 pages, accepted in Monthly Notices of the Royal Astronomical Society, Presented at the NeurIPS 2023 AI4Science Workshop

Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling

Hu Wang, Yuanhong Chen, Congbo Ma, Jodie Avery, Louise Hull, Gustavo Carneiro

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Journal ref CVPR2023

GAME-ON: Graph Attention Network based Multimodal Fusion for Fake News Detection

Mudit Dhawan, Shakshi Sharma, Aditya Kadam, Rajesh Sharma, Ponnurangam Kumaraguru

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.MM

Comments Accepted at SNAM 2024

Multimodal Approach for Harmonized System Code Prediction

Otmane Amel, Sedrick Stassin, Sidi Ahmed Mahmoudi, Xavier Siebert

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Comments Accepted for poster presentation at ESANN 2023

Journal ref Proceedings of the 31th European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2023), pp. 181-186

Token-Level Contrastive Learning with Modality-Aware Prompting for Multimodal Intent Recognition

Qianrui Zhou, Hua Xu, Hao Li, Hanlei Zhang, Xiaohan Zhang, Yifan Wang, Kai Gao

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.MM

Comments Accepted by AAAI 2024 (Main Track, Long Paper)

Enhancing Multimodal Large Language Models with Multi-instance Visual Prompt Generator for Visual Representation Enrichment

Wenliang Zhong, Wenyi Wu, Qi Li, Rob Barton, Boxin Du, Shioulin Sam, Karim Bouyarmane, Ismail Tutar, Junzhou Huang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Optimal Transport Guided Correlation Assignment for Multimodal Entity Linking

Zefeng Zhang, Jiawei Sheng, Chuang Zhang, Yunzhi Liang, Wenyuan Zhang, Siqi Wang, Tingwen Liu

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL

Comments Findings of ACL 2024

Learning Adaptive Fusion Bank for Multi-modal Salient Object Detection

Kunpeng Wang, Zhengzheng Tu, Chenglong Li, Cheng Zhang, Bin Luo

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted by TCSVT 2024

DF-DM: A foundational process model for multimodal data fusion in the artificial intelligence era

David Restrepo, Chenwei Wu, Constanza Vásquez-Venegas, Luis Filipe Nakayama, Leo Anthony Celi, Diego M López

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Comments 6 figures, 5 tables

RGB-T Object Detection via Group Shuffled Multi-receptive Attention and Multi-modal Supervision

Jinzhong Wang, Xuetao Tian, Shun Dai, Tao Zhuo, Haorui Zeng, Hongjuan Liu, Jiaqi Liu, Xiuwei Zhang, Yanning Zhang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Attention-based sequential recommendation system using multimodal data

Hyungtaik Oh, Wonkeun Jo, Dongil Kim

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Comments 18 pages, 4 figures, preprinted

SleepFM: Multi-modal Representation Learning for Sleep Across Brain Activity, ECG and Respiratory Signals

Rahul Thapa, Bryan He, Magnus Ruud Kjaer, Hyatt Moore, Gauri Ganjoo, Emmanuel Mignot, James Zou

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.AI

Lumen: Unleashing Versatile Vision-Centric Capabilities of Large Multimodal Models

Yang Jiao, Shaoxiang Chen, Zequn Jie, Jingjing Chen, Lin Ma, Yu-Gang Jiang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Comments Technical Report

Mastering Robot Manipulation with Multimodal Prompts through Pretraining and Multi-task Fine-tuning

Jiachen Li, Qiaozi Gao, Michael Johnston, Xiaofeng Gao, Xuehai He, Suhaila Shakiah, Hangjie Shi, Reza Ghanadan, William Yang Wang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.AI

Comments Accepted by ICML 2024. Project page: https://midas-icml.github.io

PM2: A New Prompting Multi-modal Model Paradigm for Few-shot Medical Image Classification

Zhenwei Wang, Qiule Sun, Bingbing Zhang, Pengfei Wang, Jianxin Zhang, Qiang Zhang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, Yueting Zhuang

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Comments Accepted by ICLR 2024 (Spotlight)

Multimodal Object Detection via Probabilistic a priori Information Integration

Hafsa El Hafyani, Bastien Pasdeloup, Camille Yver, Pierre Romenteau

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

U3M: Unbiased Multiscale Modal Fusion Model for Multimodal Semantic Segmentation

Bingyu Li, Da Zhang, Zhiyuan Zhao, Junyu Gao, Xuelong Li

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

Co-Occ: Coupling Explicit Feature Fusion with Volume Rendering Regularization for Multi-Modal 3D Semantic Occupancy Prediction

Jingyi Pan, Zipeng Wang, Lin Wang

专题命中 多模态训练与对齐 ：multi-modal(title,abstract)；分类 cs.CV

Comments Accepted by IEEE Robotics and Automation Letters (RA-L)

Journal ref IEEE Robotics and Automation Letters, Volume 9 Issue 6, 5687 - 5694, June 2024

Comprehensive Multimodal Deep Learning Survival Prediction Enabled by a Transformer Architecture: A Multicenter Study in Glioblastoma

Ahmed Gomaa, Yixing Huang, Amr Hagag, Charlotte Schmitter, Daniel Höfler, Thomas Weissmann, Katharina Breininger, Manuel Schmidt, Jenny Stritzelberger, Daniel Delev, Roland Coras, Arnd Dörfler, Oliver Schnell, Benjamin Frey, Udo S. Gaipl, Sabine Semrau, Christoph Bert, Rainer Fietkau, Florian Putz

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CV

CofiPara: A Coarse-to-fine Paradigm for Multimodal Sarcasm Target Identification with Large Multimodal Models

Hongzhan Lin, Zixin Chen, Ziyang Luo, Mingfei Cheng, Jing Ma, Guang Chen

专题命中 多模态训练与对齐 ：multimodal(title,abstract)；分类 cs.CL

Comments ACL 2024

IHC Matters: Incorporating IHC analysis to H&E Whole Slide Image Analysis for Improved Cancer Grading via Two-stage Multimodal Bilinear Pooling Fusion

Jun Wang, Yu Mao, Yufei Cui, Nan Guan, Chun Jason Xue

专题命中 多模态训练与对齐 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV