多模态大模型 - arXivDaily 专题

2107.00135 2022-12-02 cs.CV 83%

Attention Bottlenecks for Multimodal Fusion

Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, Chen Sun

专题命中多模态训练与对齐：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments Published at NeurIPS 2021. Note this version updates numbers due to a bug in the AudioSet mAP calculation in Table 1 (last row)

2205.03521 2022-12-01 cs.CL cs.AI cs.CV cs.LG cs.MM 83%

Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction

Xiang Chen, Ningyu Zhang, Lei Li, Yunzhi Yao, Shumin Deng, Chuanqi Tan, Fei Huang, Luo Si, Huajun Chen

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by NAACL 2022

2209.03126 2022-11-23 cs.MM cs.AI cs.CL cs.CV cs.LG 83%

DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention

Shunsuke Kitada, Yuki Iwazaki, Riku Togashi, Hitoshi Iyatomi

专题命中多模态训练与对齐：multi-modal(title)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 12 pages, 3 figures. Accepted by IEEE Access on Nov. 3, 2022

Journal ref in IEEE Access, vol. 10, pp. 120023-120034, 2022

1706.00999 2022-11-11 cs.CV 83%

Order embeddings and character-level convolutions for multimodal alignment

Jônatas Wehrmann, Anderson Mattjie, Rodrigo C. Barros

专题命中多模态训练与对齐：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV

Comments 7 pages, 5 figures, submitted to Pattern Recognition Letters

Journal ref Pattern Recognition Letters, vol. 102, 15, January 2018

2211.04331 2022-11-09 cs.CV 83%

Multi-Stage Based Feature Fusion of Multi-Modal Data for Human Activity Recognition

Hyeongju Choi, Apoorva Beedu, Harish Haresamudram, Irfan Essa

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

2210.06778 2022-11-01 cs.CV 83%

X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation

Shubhankar Borse, Marvin Klingner, Varun Ravi Kumar, Hong Cai, Abdulaziz Almuzairee, Senthil Yogamani, Fatih Porikli

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted to WACV 2023

2210.09615 2022-10-19 cs.CV 83%

Homogeneous Multi-modal Feature Fusion and Interaction for 3D Object Detection

Xin Li, Botian Shi, Yuenan Hou, Xingjiao Wu, Tianlong Ma, Yikang Li, Liang He

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ECCV 2022

2202.13304 2022-10-17 eess.IV cs.CV 83%

DXM-TransFuse U-net: Dual Cross-Modal Transformer Fusion U-net for Automated Nerve Identification

Baijun Xie, Gary Milam, Bo Ning, Jaepyeong Cha, Chung Hyuk Park

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Journal ref Computerized Medical Imaging and Graphics, 2022-07-01, Volume 99, Article 102090

2210.06716 2022-10-14 cs.CL 83%

Low-resource Neural Machine Translation with Cross-modal Alignment

Zhe Yang, Qingkai Fang, Yang Feng

专题命中多模态训练与对齐：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CL

Comments Accepted to EMNLP 2022

2208.10771 2022-08-24 cs.CV 83%

Learning an Efficient Multimodal Depth Completion Model

Dewang Hou, Yuanyuan Du, Kai Zhao, Yang Zhao

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments To appear in ECCV 2022 workshop and codes are available at https://github.com/dwHou/EMDC-PyTorch

2207.03305 2022-07-12 cs.AI 83%

Multimodal E-Commerce Product Classification Using Hierarchical Fusion

Tsegaye Misikir Tashu, Sara Fattouh, Peter Kiss, Tomas Horvath

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.AI

2206.11826 2022-06-27 cs.CV 83%

Toward Clinically Assisted Colorectal Polyp Recognition via Structured Cross-modal Representation Consistency

Weijie Ma, Ye Zhu, Ruimao Zhang, Jie Yang, Yiwen Hu, Zhen Li, Li Xiang

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Early Accepted by MICCAI 2022

2012.08256 2022-05-11 cs.MM 83%

A Deep Multi-Level Attentive network for Multimodal Sentiment Analysis

Ashima Yadav, Dinesh Kumar Vishwakarma

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.MM

Comments 11 pages, 7 figures

Journal ref ACM Transactions on Multimedia Computing, Communications, and Applications, 2022

2205.01818 2022-05-06 cs.LG cs.AI cs.CL cs.CV eess.AS 83%

i-Code: An Integrative and Composable Multimodal Learning Framework

Ziyi Yang, Yuwei Fang, Chenguang Zhu, Reid Pryzant, Dongdong Chen, Yu Shi, Yichong Xu, Yao Qian, Mei Gao, Yi-Ling Chen, Liyang Lu, Yujia Xie, Robert Gmyr, Noel Codella, Naoyuki Kanda, Bin Xiao, Lu Yuan, Takuya Yoshioka, Michael Zeng, Xuedong Huang

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2204.09848 2022-04-22 cs.CV 83%

Weakly Aligned Feature Fusion for Multimodal Object Detection

Lu Zhang, Zhiyong Liu, Xiangyu Zhu, Zhan Song, Xu Yang, Zhen Lei, Hong Qiao

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Journal extension of the previous conference paper arXiv:1901.02645, see IEEE page https://ieeexplore.ieee.org/document/9523596

2203.09994 2022-03-21 cs.CL cs.LG 83%

Graph-Text Multi-Modal Pre-training for Medical Representation Learning

Sungjin Park, Seongsu Bae, Jiho Kim, Tackeun Kim, Edward Choi

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments To appear in Proceedings of the Conference on Health, Inference, and Learning (CHIL 2022)

2203.03850 2022-03-09 cs.CL cs.PL cs.SE 83%

UniXcoder: Unified Cross-Modal Pre-training for Code Representation

Daya Guo, Shuai Lu, Nan Duan, Yanlin Wang, Ming Zhou, Jian Yin

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CL

Comments Published in ACL 2022

2202.10454 2022-02-23 cs.LG cs.AI 83%

A Novel Anomaly Detection Method for Multimodal WSN Data Flow via a Dynamic Graph Neural Network

Qinghao Zhang, Miao Ye, Hongbing Qiu, Yong Wang, Xiaofang Deng

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.AI

2201.09828 2022-01-25 cs.LG cs.CV 83%

MMLatch: Bottom-up Top-down Fusion for Multimodal Sentiment Analysis

Georgios Paraskevopoulos, Efthymios Georgiou, Alexandros Potamianos

专题命中多模态训练与对齐：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted, ICASSP 2022

2006.11405 2021-12-14 cs.CV cs.CL cs.LG cs.MM cs.SD eess.AS 83%

M2P2: Multimodal Persuasion Prediction using Adaptive Fusion

Chongyang Bai, Haipeng Chen, Srijan Kumar, Jure Leskovec, V. S. Subrahmanian

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments published in IEEE Trans. on Multimedia 2021

2107.07502 2021-11-11 cs.LG cs.AI cs.CL cs.CV cs.MM 83%

MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

Paul Pu Liang, Yiwei Lyu, Xiang Fan, Zetian Wu, Yun Cheng, Jason Wu, Leslie Chen, Peter Wu, Michelle A. Lee, Yuke Zhu, Ruslan Salakhutdinov, Louis-Philippe Morency

专题命中多模态训练与对齐：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2021 Datasets and Benchmarks Track. Code: https://github.com/pliang279/MultiBench and Website: https://cmu-multicomp-lab.github.io/multibench/

2104.12673 2021-10-18 cs.CV 83%

Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data

Xuhui Jia, Kai Han, Yukun Zhu, Bradley Green

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments ICCV 2021

2109.12946 2021-09-28 cs.CV 83%

Fusion-GCN: Multimodal Action Recognition using Graph Convolutional Networks

Michael Duhme, Raphael Memmesheimer, Dietrich Paulus

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 18 pages, 6 figures, 3 tables, GCPR 2021

2109.05178 2021-09-14 cs.CL 83%

College Student Retention Risk Analysis From Educational Database using Multi-Task Multi-Modal Neural Fusion

Mohammad Arif Ul Alam

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CL

Comments Submitted to 36th AAAI Conference on Artificial Intelligence (AAAI) 2022

2108.12899 2021-08-31 cs.CL cs.LG 83%

Fine-Grained Chemical Entity Typing with Multimodal Knowledge Representation

Chenkai Sun, Weijiang Li, Jinfeng Xiao, Nikolaus Nova Parulian, ChengXiang Zhai, Heng Ji

专题命中多模态训练与对齐：multimodal(title)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CL

2106.12449 2021-08-11 cs.CV 83%

FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection

Shaoqing Xu, Dingfu Zhou, Jin Fang, Junbo Yin, Zhou Bin, Liangjun Zhang

专题命中多模态训练与对齐：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by ITSC 2021

2107.06652 2021-08-09 cs.CV 83%

Self-Supervised Multi-Modal Alignment for Whole Body Medical Imaging

Rhydian Windsor, Amir Jamaludin, Timor Kadir, Andrew Zisserman

专题命中多模态训练与对齐：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted as a full paper to MICCAI 2021. Code will be made publicly available before September 27th 2021

2107.00206 2021-07-02 cs.LG cs.CV 83%

Multi-modal Graph Learning for Disease Prediction

Shuai Zheng, Zhenfeng Zhu, Zhizhe Liu, Zhenyu Guo, Yang Liu, Yao Zhao

专题命中多模态训练与对齐：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 10 pages, 4 figures, 2 tables

2106.05438 2021-06-11 cs.CV 83%

Cross-Modal Discrete Representation Learning

Alexander H. Liu, SouYoung Jin, Cheng-I Jeff Lai, Andrew Rouditchenko, Aude Oliva, James Glass

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Preprint

2011.01631 2021-03-10 cs.LG cs.MM 83%

Robust Latent Representations via Cross-Modal Translation and Alignment

Vandana Rajan, Alessio Brutti, Andrea Cavallaro

专题命中多模态训练与对齐：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.MM

Journal ref ICASSP 2021

1. 多模态训练与对齐 6872 篇

Attention Bottlenecks for Multimodal Fusion

Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction

DM$^2$S$^2$: Deep Multi-Modal Sequence Sets with Hierarchical Modality Attention

Order embeddings and character-level convolutions for multimodal alignment

Multi-Stage Based Feature Fusion of Multi-Modal Data for Human Activity Recognition

X-Align: Cross-Modal Cross-View Alignment for Bird's-Eye-View Segmentation

Homogeneous Multi-modal Feature Fusion and Interaction for 3D Object Detection

DXM-TransFuse U-net: Dual Cross-Modal Transformer Fusion U-net for Automated Nerve Identification

Low-resource Neural Machine Translation with Cross-modal Alignment

Learning an Efficient Multimodal Depth Completion Model

Multimodal E-Commerce Product Classification Using Hierarchical Fusion

Toward Clinically Assisted Colorectal Polyp Recognition via Structured Cross-modal Representation Consistency

A Deep Multi-Level Attentive network for Multimodal Sentiment Analysis

i-Code: An Integrative and Composable Multimodal Learning Framework

Weakly Aligned Feature Fusion for Multimodal Object Detection

Graph-Text Multi-Modal Pre-training for Medical Representation Learning

UniXcoder: Unified Cross-Modal Pre-training for Code Representation

A Novel Anomaly Detection Method for Multimodal WSN Data Flow via a Dynamic Graph Neural Network

MMLatch: Bottom-up Top-down Fusion for Multimodal Sentiment Analysis

M2P2: Multimodal Persuasion Prediction using Adaptive Fusion

MultiBench: Multiscale Benchmarks for Multimodal Representation Learning

Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data

Fusion-GCN: Multimodal Action Recognition using Graph Convolutional Networks

College Student Retention Risk Analysis From Educational Database using Multi-Task Multi-Modal Neural Fusion

Fine-Grained Chemical Entity Typing with Multimodal Knowledge Representation

FusionPainting: Multimodal Fusion with Adaptive Attention for 3D Object Detection

Self-Supervised Multi-Modal Alignment for Whole Body Medical Imaging

Multi-modal Graph Learning for Disease Prediction

Cross-Modal Discrete Representation Learning

Robust Latent Representations via Cross-Modal Translation and Alignment