多模态大模型 - arXivDaily 专题

2303.07667 2023-06-13 cs.SD 82%

Improving Music Genre Classification from Multi-Modal Properties of Music and Genre Correlations Perspective

Ganghui Ru, Xulong Zhang, Jianzong Wang, Ning Cheng, Jing Xiao

专题命中音频语音多模态：multi-modal(title,abstract)；cross-modal(abstract)

Comments Accepted by ICASSP 2023

2305.19458 2023-06-01 cs.SD cs.CV cs.LG cs.MM eess.AS 82%

A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition

Shentong Mo, Pedro Morgado

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

2305.14635 2023-05-26 cs.CL cs.AI cs.SD eess.AS 82%

CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation

Yan Zhou, Qingkai Fang, Yang Feng

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments ACL 2023 main conference

2305.04476 2023-05-25 eess.AS cs.CL cs.MM cs.SD 82%

AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment

Ruiqi Li, Rongjie Huang, Lichao Zhang, Jinglin Liu, Zhou Zhao

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Findings of ACL 2023

2305.02814 2023-05-05 cs.MM cs.AI cs.CV cs.HC 82%

Noise-Resistant Multimodal Transformer for Emotion Recognition

Yuanyuan Liu, Haoyu Zhang, Yibing Zhan, Zijing Chen, Guanghao Yin, Lin Wei, Zhe Chen

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2211.03643 2023-05-04 cs.SD cs.CL cs.MM eess.AS 82%

Egocentric Audio-Visual Noise Suppression

Roshan Sharma, Weipeng He, Ju Lin, Egor Lakomkin, Yang Liu, Kaustubh Kalgaonkar

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted to ICASSP 2023

2210.07839 2023-04-13 cs.MM cs.CV cs.SD eess.AS 82%

Contrastive Audio-Visual Masked Autoencoder

Yuan Gong, Andrew Rouditchenko, Alexander H. Liu, David Harwath, Leonid Karlinsky, Hilde Kuehne, James Glass

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at ICLR 2023 as a notable top 25% paper. Code and pretrained models are at https://github.com/yuangongnd/cav-mae

2212.07983 2023-04-06 cs.CV cs.CL cs.LG cs.SD eess.AS 82%

Vision Transformers are Parameter-Efficient Audio-Visual Learners

Yan-Bo Lin, Yi-Lin Sung, Jie Lei, Mohit Bansal, Gedas Bertasius

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments CVPR 2023 Project Page: https://genjib.github.io/project_page/LAVISH/

2303.06904 2023-03-14 cs.CV cs.AI cs.CL 82%

Contextually-rich human affect perception using multimodal scene information

Digbalay Bose, Rajat Hebbar, Krishna Somandepalli, Shrikanth Narayanan

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2023

2207.05042 2023-02-20 cs.CV cs.MM cs.SD eess.AS eess.IV 82%

Audio-Visual Segmentation

Jinxing Zhou, Jianyuan Wang, Jiayi Zhang, Weixuan Sun, Jing Zhang, Stan Birchfield, Dan Guo, Lingpeng Kong, Meng Wang, Yiran Zhong

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments ECCV 2022; Code is available at https://github.com/OpenNLPLab/AVSBench

2212.05301 2023-02-03 cs.SD cs.CV cs.MM eess.AS 82%

Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning

Chen Chen, Yuchen Hu, Qiang Zhang, Heqing Zou, Beier Zhu, Eng Siong Chng

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by AAAI2023

2301.01456 2023-01-05 cs.CV cs.CL cs.SD eess.AS 82%

Audio-Visual Efficient Conformer for Robust Speech Recognition

Maxime Burchi, Radu Timofte

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

2301.01134 2023-01-04 cs.MM cs.CL cs.CV 82%

Ring That Bell: A Corpus and Method for Multimodal Metaphor Detection in Videos

Khalid Alnajjar, Mika Hämäläinen, Shuo Zhang

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Figlang 2022

2212.14490 2023-01-02 cs.SD cs.CL cs.MM eess.AS 82%

Multi-modal deep learning system for depression and anxiety detection

Brian Diep, Marija Stanojevic, Jekaterina Novikova

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments accepted to the PAI4MH workshop at NeurIPS 2022

2212.04408 2022-12-09 cs.CV cs.AI cs.CL cs.LG 82%

OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models

Jinze Bai, Rui Men, Hao Yang, Xuancheng Ren, Kai Dang, Yichang Zhang, Xiaohuan Zhou, Peng Wang, Sinan Tan, An Yang, Zeyu Cui, Yu Han, Shuai Bai, Wenbin Ge, Jianxin Ma, Junyang Lin, Jingren Zhou, Chang Zhou

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2212.01040 2022-12-05 cs.CV cs.MM cs.SD eess.AS 82%

Role of Audio in Audio-Visual Video Summarization

Ibrahim Shoer, Berkay Kopru, Engin Erzin

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

2212.00500 2022-12-02 cs.MM cs.CL cs.LG cs.SD eess.AS 82%

MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognition

Xiaohuan Zhou, Jiaming Wang, Zeyu Cui, Shiliang Zhang, Zhijie Yan, Jingren Zhou, Chang Zhou

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Submitted to ICASSP 2023

2211.04368 2022-11-09 cs.CL cs.CV cs.SD eess.AS 82%

A Multimodal Approach for Dementia Detection from Spontaneous Speech with Tensor Fusion Layer

Loukas Ilias, Dimitris Askounis, John Psarras

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments 2022 IEEE-EMBS International Conference on Biomedical and Health Informatics (BHI) - Oral Presentation

2211.02077 2022-11-07 cs.CV cs.AI cs.LG cs.MM 82%

Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization

Junru Wu, Yi Liang, Feng Han, Hassan Akbari, Zhangyang Wang, Cong Yu

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted at NeurIPS 2022

2209.04093 2022-10-27 cs.CV cs.MM cs.SD eess.AS 82%

Learning Audio-Visual embedding for Person Verification in the Wild

Peiwen Sun, Shanshan Zhang, Zishan Liu, Yougen Yuan, Taotao Zhang, Honggang Zhang, Pengfei Hu

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

2203.04114 2022-09-02 cs.MM cs.CV cs.SD eess.AS 82%

A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification

Qing Wang, Jun Du, Siyuan Zheng, Yunqing Li, Yajian Wang, Yuzhong Wu, Hu Hu, Chao-Han Huck Yang, Sabato Marco Siniscalchi, Yannan Wang, Chin-Hui Lee

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 5 pages, 1 figure

2204.03063 2022-08-29 cs.MM cs.CV cs.IR cs.SD eess.AS 82%

Late multimodal fusion for image and audio music transcription

María Alfaro-Contreras, Jose J. Valero-Mas, José M. Iñesta, Jorge Calvo-Zaragoza

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

2111.14448 2022-07-19 cs.CV cs.MM eess.AS 82%

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

Eric Zhongcong Xu, Zeyang Song, Satoshi Tsutsui, Chao Feng, Mang Ye, Mike Zheng Shou

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments ACMMM 2022

2204.14057 2022-05-30 cs.SD cs.CV cs.MM eess.AS 82%

Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast

Boqing Zhu, Kele Xu, Changjian Wang, Zheng Qin, Tao Sun, Huaimin Wang, Yuxing Peng

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 8 pages, 4 figures. Accepted by IJCAI-2022

2205.08383 2022-05-18 cs.LG cs.AI cs.CL cs.CV 82%

Bias and Fairness on Multimodal Emotion Detection Algorithms

Matheus Schmitz, Rehan Ahmed, Jimi Cao

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2205.05072 2022-05-11 cs.CV cs.MM cs.SD eess.AS 82%

Learning Visual Styles from Audio-Visual Associations

Tingle Li, Yichen Liu, Andrew Owens, Hang Zhao

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

2204.01977 2022-04-11 cs.SD cs.CV cs.MM eess.AS 82%

Audio-visual multi-channel speech separation, dereverberation and recognition

Guinan Li, Jianwei Yu, Jiajun Deng, Xunying Liu, Helen Meng

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by ICASSP 2022

2203.07990 2022-03-16 cs.MM cs.AI cs.CL 82%

UofA-Truth at Factify 2022 : Transformer And Transfer Learning Based Multi-Modal Fact-Checking

Abhishek Dhankar, Osmar R. Zaïane, Francois Bolduc

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

2203.02013 2022-03-07 cs.LG cs.AI cs.CL cs.CV 82%

DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations

Yiwei Lyu, Paul Pu Liang, Zihao Deng, Ruslan Salakhutdinov, Louis-Philippe Morency

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Code available at https://github.com/lvyiwei1/DIME

2108.05080 2022-03-02 cs.CV cs.MM cs.SD eess.AS 82%

FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset

Hasam Khalid, Shahroz Tariq, Minha Kim, Simon S. Woo

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Part of Proceedings of the Neural Information Processing Systems Track on Datasets and Benchmarks (NeurIPS Datasets and Benchmarks 2021)

1. 音频语音多模态 4557 篇

Improving Music Genre Classification from Multi-Modal Properties of Music and Genre Correlations Perspective

A Unified Audio-Visual Learning Framework for Localization, Separation, and Recognition

CMOT: Cross-modal Mixup via Optimal Transport for Speech Translation

AlignSTS: Speech-to-Singing Conversion via Cross-Modal Alignment

Noise-Resistant Multimodal Transformer for Emotion Recognition

Egocentric Audio-Visual Noise Suppression

Contrastive Audio-Visual Masked Autoencoder

Vision Transformers are Parameter-Efficient Audio-Visual Learners

Contextually-rich human affect perception using multimodal scene information

Audio-Visual Segmentation

Leveraging Modality-specific Representations for Audio-visual Speech Recognition via Reinforcement Learning

Audio-Visual Efficient Conformer for Robust Speech Recognition

Ring That Bell: A Corpus and Method for Multimodal Metaphor Detection in Videos

Multi-modal deep learning system for depression and anxiety detection

OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models

Role of Audio in Audio-Visual Video Summarization

MMSpeech: Multi-modal Multi-task Encoder-Decoder Pre-training for Speech Recognition

A Multimodal Approach for Dementia Detection from Spontaneous Speech with Tensor Fusion Layer

Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonization

Learning Audio-Visual embedding for Person Verification in the Wild

A study on joint modeling and data augmentation of multi-modalities for audio-visual scene classification

Late multimodal fusion for image and audio music transcription

AVA-AVD: Audio-Visual Speaker Diarization in the Wild

Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast

Bias and Fairness on Multimodal Emotion Detection Algorithms

Learning Visual Styles from Audio-Visual Associations

Audio-visual multi-channel speech separation, dereverberation and recognition

UofA-Truth at Factify 2022 : Transformer And Transfer Learning Based Multi-Modal Fact-Checking

DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations

FakeAVCeleb: A Novel Audio-Video Multimodal Deepfake Dataset