多模态大模型 - arXivDaily 专题

2402.17187 2024-04-18 eess.IV cs.CV 83%

PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction

Zhaoxin Guo, Zhipeng Wang, Ruiquan Ge, Jianxun Yu, Feiwei Qin, Yuan Tian, Yuqing Peng, Yonghong Li, Changmiao Wang

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

2307.15432 2024-04-16 cs.CL 83%

CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion Recognition

Jiang Li, Xiaoping Wang, Yingjian Liu, Zhigang Zeng

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL

Comments Accepted by IEEE Transactions on Affective Computing (TAFFC)

2311.10707 2024-04-02 cs.LG cs.CV 83%

Multimodal Representation Learning by Alternating Unimodal Adaptation

Xiaohui Zhang, Jaehong Yoon, Mohit Bansal, Huaxiu Yao

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by CVPR 2024

2403.16558 2024-04-01 cs.CV 83%

Elysium: Exploring Object-level Perception in Videos via MLLM

Han Wang, Yanjie Wang, Yongjie Ye, Yuxiang Nie, Can Huang

专题命中多模态训练与对齐：MLLM(title,abstract)；multi-modal(abstract)；分类 cs.CV

2403.19203 2024-03-30 eess.IV cs.CV 83%

Single-Shared Network with Prior-Inspired Loss for Parameter-Efficient Multi-Modal Imaging Skin Lesion Classification

Peng Tang, Tobias Lasser

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments This paper have submitted to Journal for review

2306.14565 2024-03-21 cs.CV cs.AI cs.CE cs.CL cs.MM 83%

Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

Fuxiao Liu, Kevin Lin, Linjie Li, Jianfeng Wang, Yaser Yacoob, Lijuan Wang

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 40 pages, 32 figures, ICLR 2024

2403.09672 2024-03-18 cs.CV cs.LG 83%

COMPRER: A Multimodal Multi-Objective Pretraining Framework for Enhanced Medical Image Representation

Guy Lutsker, Hagai Rossman, Nastya Godiva, Eran Segal

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2403.07372 2024-03-13 cs.CV 83%

Eliminating Cross-modal Conflicts in BEV Space for LiDAR-Camera 3D Object Detection

Jiahui Fu, Chen Gao, Zitian Wang, Lirong Yang, Xiaofei Wang, Beipeng Mu, Si Liu

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by ICRA 2024

2311.17911 2024-03-13 cs.CV 83%

OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation

Qidong Huang, Xiaoyi Dong, Pan Zhang, Bin Wang, Conghui He, Jiaqi Wang, Dahua Lin, Weiming Zhang, Nenghai Yu

专题命中多模态训练与对齐：multi-modal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments CVPR 2024, code is available at https://github.com/shikiw/OPERA

2403.03707 2024-03-07 cs.CV 83%

Multi-Grained Cross-modal Alignment for Learning Open-vocabulary Semantic Segmentation from Text Supervision

Yajie Liu, Pu Ge, Qingjie Liu, Di Huang

专题命中多模态训练与对齐：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments 17 pages, 8 figures

2402.19298 2024-03-06 cs.CV 83%

Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing

Xun Lin, Shuai Wang, Rizhao Cai, Yizhong Liu, Ying Fu, Zitong Yu, Wenzhong Tang, Alex Kot

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepeted by CVPR 2024

2402.15858 2024-02-27 cs.CV cs.DC 83%

FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathology

Yuanzhe Peng, Jieming Bian, Jie Xu

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Journal ref 2024 International Conference on Acoustics, Speech and Signal Processing (ICASSP 2024)

2310.09503 2024-01-26 cs.CV 83%

JM3D & JM3D-LLM: Elevating 3D Understanding with Joint Multi-modal Cues

Jiayi Ji, Haowei Wang, Changli Wu, Yiwei Ma, Xiaoshuai Sun, Rongrong Ji

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 16 pages, 4 figures, 10 tables, 3D understanding

2308.02982 2024-01-26 cs.CV 83%

Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation

Haowei Wang, Jiji Tang, Jiayi Ji, Xiaoshuai Sun, Rongsheng Zhang, Yiwei Ma, Minda Zhao, Lincheng Li, zeng zhao, Tangjie Lv, Rongrong Ji

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments ACM MM 2023, 3D Understanding, JM3D

2401.09638 2024-01-19 eess.IV cs.CV cs.LG 83%

Automatic 3D Multi-modal Ultrasound Segmentation of Human Placenta using Fusion Strategies and Deep Learning

Sonit Singh, Gordon Stevenson, Brendan Mein, Alec Welsh, Arcot Sowmya

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

2401.08123 2024-01-17 cs.CV 83%

The Devil is in the Details: Boosting Guided Depth Super-Resolution via Rethinking Cross-Modal Alignment and Aggregation

Xinni Jiang, Zengsheng Kuang, Chunle Guo, Ruixun Zhang, Lei Cai, Xiao Fan, Chongyi Li

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

2303.16818 2024-01-09 cs.CV 83%

SimDistill: Simulated Multi-modal Distillation for BEV 3D Object Detection

Haimei Zhao, Qiming Zhang, Shanshan Zhao, Zhe Chen, Jing Zhang, Dacheng Tao

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by AAAI 2024

2312.16648 2023-12-29 cs.RO cs.CV 83%

LIP-Loc: LiDAR Image Pretraining for Cross-Modal Localization

Sai Shubodh Puligilla, Mohammad Omama, Husain Zaidi, Udit Singh Parihar, Madhava Krishna

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments To be presented at WACV-W 2024. Project page: https://shubodhs.ai/liploc

2312.15645 2023-12-27 cs.CL 83%

Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment

Rui Zhao, Liang Zhang, Biao Fu, Cong Hu, Jinsong Su, Yidong Chen

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CL

Comments Accepted as conference paper by AAAI24. The code and models are available at https://github.com/rzhao-zhsq/CV-SLT

2211.13929 2023-12-27 cs.CV 83%

XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning

Pritam Sarkar, Ali Etemad

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments AAAI 2024

2312.14471 2023-12-25 cs.CV 83%

Prototype-based Cross-Modal Object Tracking

Lei Liu, Chenglong Li, Futian Wang, Longfeng Shen, Jin Tang

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments In Peer Review

2312.14446 2023-12-25 cs.CV 83%

Cross-Modal Object Tracking via Modality-Aware Fusion Network and A Large-Scale Dataset

Lei Liu, Mengya Zhang, Cheng Li, Chenglong Li, Jin Tang

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments In Peer Review

2310.05804 2023-12-15 cs.AI cs.CL cs.CV cs.MM 83%

Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis

Haoyu Zhang, Yu Wang, Guanghao Yin, Kejun Liu, Yuanyuan Liu, Tianshu Yu

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Published in EMNLP 2023

Journal ref Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

2311.12793 2023-11-29 cs.CV 83%

ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, Dahua Lin

专题命中多模态训练与对齐：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments Project: https://ShareGPT4V.github.io

2203.04838 2023-11-27 cs.CV cs.RO eess.IV 83%

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, Rainer Stiefelhagen

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted to IEEE Transactions on Intelligent Transportation Systems (T-ITS). The source code of CMX is publicly available at https://github.com/huaaaliu/RGBX_Semantic_Segmentation

2311.14315 2023-11-27 cs.AI 83%

Robust Domain Misinformation Detection via Multi-modal Feature Alignment

Hui Liu, Wenya Wang, Hao Sun, Anderson Rocha, Haoliang Li

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments Accepted by TIFS 2023

2311.03106 2023-11-07 cs.CV 83%

Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding

Shengkai Sun, Daizong Liu, Jianfeng Dong, Xiaoye Qu, Junyu Gao, Xun Yang, Xun Wang, Meng Wang

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted by ACM MM 2023. The code is available at https://github.com/HuiGuanLab/UmURL

2310.16754 2023-10-31 cs.CV 83%

CAD -- Contextual Multi-modal Alignment for Dynamic AVQA

Asmar Nadeem, Adrian Hilton, Robert Dawes, Graham Thomas, Armin Mustafa

专题命中多模态训练与对齐：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted to IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) 2024

2310.14143 2023-10-24 cs.CV cs.LG 83%

MMTF-DES: A Fusion of Multimodal Transformer Models for Desire, Emotion, and Sentiment Analysis of Social Media Data

Abdul Aziz, Nihad Karim Chowdhury, Muhammad Ashad Kabir, Abu Nowshed Chy, Md. Jawad Siddique

专题命中多模态训练与对齐：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments 28 pages, 4 figures

2310.12100 2023-10-19 cs.CL cs.AI cs.CV cs.LG cs.MM 83%

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling

Yaqing Wang, Jialin Wu, Tanmaya Dabral, Jiageng Zhang, Geoff Brown, Chun-Ta Lu, Frederick Liu, Yi Liang, Bo Pang, Michael Bendersky, Radu Soricut

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

1. 多模态训练与对齐 6872 篇

PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction

CFN-ESA: A Cross-Modal Fusion Network with Emotion-Shift Awareness for Dialogue Emotion Recognition

Multimodal Representation Learning by Alternating Unimodal Adaptation

Elysium: Exploring Object-level Perception in Videos via MLLM

Single-Shared Network with Prior-Inspired Loss for Parameter-Efficient Multi-Modal Imaging Skin Lesion Classification

Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning

COMPRER: A Multimodal Multi-Objective Pretraining Framework for Enhanced Medical Image Representation

Eliminating Cross-modal Conflicts in BEV Space for LiDAR-Camera 3D Object Detection

OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation

Multi-Grained Cross-modal Alignment for Learning Open-vocabulary Semantic Segmentation from Text Supervision

Suppress and Rebalance: Towards Generalized Multi-Modal Face Anti-Spoofing

FedMM: Federated Multi-Modal Learning with Modality Heterogeneity in Computational Pathology

JM3D & JM3D-LLM: Elevating 3D Understanding with Joint Multi-modal Cues

Beyond First Impressions: Integrating Joint Multi-modal Cues for Comprehensive 3D Representation

Automatic 3D Multi-modal Ultrasound Segmentation of Human Placenta using Fusion Strategies and Deep Learning

The Devil is in the Details: Boosting Guided Depth Super-Resolution via Rethinking Cross-Modal Alignment and Aggregation

SimDistill: Simulated Multi-modal Distillation for BEV 3D Object Detection

LIP-Loc: LiDAR Image Pretraining for Cross-Modal Localization

Conditional Variational Autoencoder for Sign Language Translation with Cross-Modal Alignment

XKD: Cross-modal Knowledge Distillation with Domain Alignment for Video Representation Learning

Prototype-based Cross-Modal Object Tracking

Cross-Modal Object Tracking via Modality-Aware Fusion Network and A Large-Scale Dataset

Learning Language-guided Adaptive Hyper-modality Representation for Multimodal Sentiment Analysis

ShareGPT4V: Improving Large Multi-Modal Models with Better Captions

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Robust Domain Misinformation Detection via Multi-modal Feature Alignment

Unified Multi-modal Unsupervised Representation Learning for Skeleton-based Action Understanding

CAD -- Contextual Multi-modal Alignment for Dynamic AVQA

MMTF-DES: A Fusion of Multimodal Transformer Models for Desire, Emotion, and Sentiment Analysis of Social Media Data

Non-Intrusive Adaptation: Input-Centric Parameter-efficient Fine-Tuning for Versatile Multimodal Modeling