多模态大模型 - arXivDaily 专题

2403.13537 2024-03-21 cs.CL cs.AI cs.CV cs.LG 82%

What explains the success of cross-modal fine-tuning with ORCA?

Paloma García-de-Herreros, Vagrant Gautam, Philipp Slusallek, Dietrich Klakow, Marius Mosbach

专题命中多模态训练与对齐：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2403.10279 2024-03-18 cs.CY 82%

Emotion-Aware Multimodal Fusion for Meme Emotion Detection

Shivam Sharma, Ramaneswaran S, Md. Shad Akhtar, Tanmoy Chakraborty

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)

Comments Accepted to IEEE Transactions on Affective Computing

2402.19407 2024-03-01 cs.IR 82%

MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation

Jinfeng Xu, Zheyu Chen, Shuo Yang, Jinze Li, Hewei Wang, Edith C. -H. Ngai

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)

2402.15444 2024-02-26 cs.AI cs.CL cs.LG cs.MM 82%

Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion

Yichi Zhang, Zhuo Chen, Lei Liang, Huajun Chen, Wen Zhang

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted by LREC-COLING 2024

2402.05294 2024-02-09 cs.LG cs.AI cs.CL cs.CV 82%

Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection

Pramit Saha, Divyanshu Mishra, Felix Wagner, Konstantinos Kamnitsas, J. Alison Noble

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 42 pages

2401.06827 2024-01-24 cs.CV cs.AI cs.CL 82%

APLe: Token-Wise Adaptive for Multi-Modal Prompt Learning

Guiming Cao, Kaize Shi, Hong Fu, Huaiwen Zhang, Guandong Xu

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 7 pages,3 figures

2312.10386 2023-12-19 cs.LG 82%

RedCore: Relative Advantage Aware Cross-modal Representation Learning for Missing Modalities with Imbalanced Missing Rates

Jun Sun, Xinxin Zhang, Shoukang Han, Yu-ping Ruan, Taihao Li

专题命中多模态训练与对齐：cross-modal(title,abstract)；multimodal(abstract)

2312.03231 2023-12-07 cs.LG cs.AI cs.CV cs.HC eess.AS 82%

Deep Multimodal Fusion for Surgical Feedback Classification

Rafal Kocielnik, Elyssa Y. Wong, Timothy N. Chu, Lydia Lin, De-An Huang, Jiayun Wang, Anima Anandkumar, Andrew J. Hung

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Journal ref Published in Proceedings of Machine Learning for Health 2024

2305.13583 2023-11-14 cs.CL cs.MM eess.AS eess.IV 82%

Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition

Yaoting Wang, Yuanchao Li, Paul Pu Liang, Louis-Philippe Morency, Peter Bell, Catherine Lai

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments *First two authors contributed equally

2310.16822 2023-10-26 cs.CL cs.AI cs.MM 82%

Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction

Xuming Hu, Junzhe Chen, Aiwei Liu, Shiao Meng, Lijie Wen, Philip S. Yu

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments Accepted to ACM Multimedia 2023

2309.14398 2023-09-28 cs.LG cs.AI cs.CL eess.AS 82%

Seeing and hearing what has not been said; A multimodal client behavior classifier in Motivational Interviewing with interpretable fusion

Lucie Galland, Catherine Pelachaud, Florian Pecune

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 9 pages, 7 figures

2205.11501 2023-09-18 cs.CV cs.AI cs.CL 82%

VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering

Yanan Wang, Michihiro Yasunaga, Hongyu Ren, Shinya Wada, Jure Leskovec

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at ICCV 2023

2309.07120 2023-09-14 cs.CL cs.AI cs.CV cs.CY cs.LG 82%

Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics

Haoqin Tu, Bingchen Zhao, Chen Wei, Cihang Xie

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2309.02106 2023-09-06 cs.CL cs.AI cs.LG eess.AS 82%

Leveraging Label Information for Multimodal Emotion Recognition

Peiying Wang, Sunlu Zeng, Junqing Chen, Lu Fan, Meng Chen, Youzheng Wu, Xiaodong He

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted by Interspeech 2023

2307.16210 2023-08-02 cs.AI cs.CV cs.LG cs.MM 82%

Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment

Zhuo Chen, Lingbing Guo, Yin Fang, Yichi Zhang, Jiaoyan Chen, Jeff Z. Pan, Yangning Li, Huajun Chen, Wen Zhang

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments International Semantic Web Conference '23 (ISWC 2023), https://github.com/zjukg/UMAEA

2307.16745 2023-08-01 cs.CV cs.AI cs.CY cs.MM 82%

Advancing Smart Malnutrition Monitoring: A Multi-Modal Learning Approach for Vital Health Parameter Estimation

Ashish Marisetty, Prathistith Raj M, Praneeth Nemani, Venkanna Udutalapally, Debanjan Das

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2307.05390 2023-07-12 cond-mat.mtrl-sci cs.LG 82%

CrysMMNet: Multimodal Representation for Crystal Property Prediction

Kishalay Das, Pawan Goyal, Seung-Cheol Lee, Satadeep Bhattacharjee, Niloy Ganguly

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)

Comments 14 pages, 4 fifures

2307.02761 2023-07-07 cs.IR 82%

Cross-Modal Content Inference and Feature Enrichment for Cold-Start Recommendation

Haokai Ma, Zhuang Qi, Xinxin Dong, Xiangxian Li, Yuze Zheng, Xiangxu Mengand Lei Meng

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)

Comments Accepted by International Joint Conference on Neural Networks 2023 (IJCNN)

2306.02137 2023-06-21 cs.LG 82%

Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection

Mengzhu Sun, Xi Zhang, Jianqiang Ma, Sihong Xie, Yazheng Liu, Philip S. Yu

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)

Journal ref IEEE Transactions on Knowledge and Data Engineering, 2023

2209.05722 2023-05-18 cs.RO 82%

GrASPE: Graph based Multimodal Fusion for Robot Navigation in Unstructured Outdoor Environments

Kasun Weerakoon, Adarsh Jagan Sathyamoorthy, Jing Liang, Tianrui Guan, Utsav Patel, Dinesh Manocha

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)

2304.08028 2023-04-18 cs.CV cs.AI cs.MM 82%

MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning

Shicai Wei, Yang Luo, Chunbo Luo

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 10 pages, 3 figures, CVPR2023

2209.15182 2023-04-12 cs.HC cs.RO 82%

Husformer: A Multi-Modal Transformer for Multi-Modal Human State Recognition

Ruiqi Wang, Wonse Jo, Dezhong Zhao, Weizheng Wang, Baijian Yang, Guohua Chen, Byung-Cheol Min

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)

2207.07278 2023-04-07 cs.CV cs.CL cs.MM 82%

Boosting Multi-Modal E-commerce Attribute Value Extraction via Unified Learning Scheme and Dynamic Range Minimization

Mengyin Liu, Chao Zhu, Hongyu Gao, Weibo Gu, Hongfa Wang, Wei Liu, Xu-cheng Yin

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

2211.15028 2023-02-21 cs.CL cs.AI cs.MM 82%

Joint Multimodal Entity-Relation Extraction Based on Edge-enhanced Graph Alignment Network and Word-pair Relation Tagging

Li Yuan, Yi Cai, Jin Wang, Qing Li

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

Comments accepted in AAAI-2023

2302.07740 2023-02-16 cs.CL cs.AI cs.CV cs.LG 82%

Team Triple-Check at Factify 2: Parameter-Efficient Large Foundation Models with Feature Representations for Multi-Modal Fact Verification

Wei-Wei Du, Hong-Wei Wu, Wei-Yao Wang, Wen-Chih Peng

专题命中多模态训练与对齐：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments AAAI-23 DeFactify 2 Workshop (1st Prize)

2302.05289 2023-02-13 cs.CV cs.AI cs.CL cs.SI 82%

Rumor Classification through a Multimodal Fusion Framework and Ensemble Learning

Abderrazek Azri, Cécile Favre, Nouria Harbi, Jérôme Darmont, Camille Noûs

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Information Systems Frontiers, 2022

2301.00254 2023-01-03 cs.MM cs.AI cs.CV 82%

Depression Diagnosis and Analysis via Multimodal Multi-order Factor Fusion

Chengbo Yuan, Qianhui Xu, Yong Luo

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2111.10974 2022-12-29 cs.CV cs.AI cs.CL 82%

Many Heads but One Brain: Fusion Brain -- a Competition and a Single Multimodal Multitask Architecture

Daria Bakshandaeva, Denis Dimitrov, Vladimir Arkhipkin, Alex Shonenkov, Mark Potanin, Denis Karachev, Andrey Kuznetsov, Anton Voronov, Vera Davydova, Elena Tutubalina, Aleksandr Petiushko

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2211.14739 2022-11-29 cs.CV cs.AI cs.CL 82%

MNER-QG: An End-to-End MRC framework for Multimodal Named Entity Recognition with Query Grounding

Meihuizi Jia, Lei Shen, Xin Shen, Lejian Liao, Meng Chen, Xiaodong He, Zhendong Chen, Jiaqi Li

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 13 pages, 6 figures, published to AAAI

2208.11064 2022-10-18 cs.LG 82%

Multi-Modal Representation Learning with Self-Adaptive Threshold for Commodity Verification

Chenchen Han, Heng Jia

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)

1. 多模态训练与对齐 6872 篇

What explains the success of cross-modal fine-tuning with ORCA?

Emotion-Aware Multimodal Fusion for Meme Emotion Detection

MENTOR: Multi-level Self-supervised Learning for Multimodal Recommendation

Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion

Examining Modality Incongruity in Multimodal Federated Learning for Medical Vision and Language-based Disease Detection

APLe: Token-Wise Adaptive for Multi-Modal Prompt Learning

RedCore: Relative Advantage Aware Cross-modal Representation Learning for Missing Modalities with Imbalanced Missing Rates

Deep Multimodal Fusion for Surgical Feedback Classification

Cross-Attention is Not Enough: Incongruity-Aware Dynamic Hierarchical Fusion for Multimodal Affect Recognition

Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction

Seeing and hearing what has not been said; A multimodal client behavior classifier in Motivational Interviewing with interpretable fusion

VQA-GNN: Reasoning with Multimodal Knowledge via Graph Neural Networks for Visual Question Answering

Sight Beyond Text: Multi-Modal Training Enhances LLMs in Truthfulness and Ethics

Leveraging Label Information for Multimodal Emotion Recognition

Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment

Advancing Smart Malnutrition Monitoring: A Multi-Modal Learning Approach for Vital Health Parameter Estimation

CrysMMNet: Multimodal Representation for Crystal Property Prediction

Cross-Modal Content Inference and Feature Enrichment for Cold-Start Recommendation

Inconsistent Matters: A Knowledge-guided Dual-consistency Network for Multi-modal Rumor Detection

GrASPE: Graph based Multimodal Fusion for Robot Navigation in Unstructured Outdoor Environments

MMANet: Margin-aware Distillation and Modality-aware Regularization for Incomplete Multimodal Learning

Husformer: A Multi-Modal Transformer for Multi-Modal Human State Recognition

Boosting Multi-Modal E-commerce Attribute Value Extraction via Unified Learning Scheme and Dynamic Range Minimization

Joint Multimodal Entity-Relation Extraction Based on Edge-enhanced Graph Alignment Network and Word-pair Relation Tagging

Team Triple-Check at Factify 2: Parameter-Efficient Large Foundation Models with Feature Representations for Multi-Modal Fact Verification

Rumor Classification through a Multimodal Fusion Framework and Ensemble Learning

Depression Diagnosis and Analysis via Multimodal Multi-order Factor Fusion

Many Heads but One Brain: Fusion Brain -- a Competition and a Single Multimodal Multitask Architecture

MNER-QG: An End-to-End MRC framework for Multimodal Named Entity Recognition with Query Grounding

Multi-Modal Representation Learning with Self-Adaptive Threshold for Commodity Verification