多模态大模型

Audio-visual scene classification via contrastive event-object alignment and semantic-based fusion

Yuanbo Hou, Bo Kang, Dick Botteldooren

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments IEEE MMSP 2022

Active Audio-Visual Separation of Dynamic Sound Sources

Sagnik Majumder, Kristen Grauman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to ECCV 2022

VoViT: Low Latency Graph-based Audio-Visual Voice Separation Transformer

Juan F. Montesinos, Venkatesh S. Kadandale, Gloria Haro

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to ECCV 2022

Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERT

Bowen Shi, Abdelrahman Mohamed, Wei-Ning Hsu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Interspeech 2022

Robust Self-Supervised Audio-Visual Speech Recognition

Bowen Shi, Wei-Ning Hsu, Abdelrahman Mohamed

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Interspeech 2022

Finding Fallen Objects Via Asynchronous Audio-Visual Integration

Chuang Gan, Yi Gu, Siyuan Zhou, Jeremy Schwartz, Seth Alter, James Traer, Dan Gutfreund, Joshua B. Tenenbaum, Josh McDermott, Antonio Torralba

专题命中 音频语音多模态 ：audio-visual(title)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments CVPR 2022. Project page: http://fallen-object.csail.mit.edu

Kaggle Competition: Cantonese Audio-Visual Speech Recognition for In-car Commands

Wenliang Dai, Samuel Cahyawijaya, Tiezheng Yu, Elham J Barezi, Pascale Fung

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CL、eess.AS

Leveraging Acoustic Contextual Representation by Audio-textual Cross-modal Learning for Conversational ASR

Kun Wei, Yike Zhang, Sining Sun, Lei Xie, Long Ma

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted by Interspeech2022

Improving Visual Speech Enhancement Network by Learning Audio-visual Affinity with Multi-head Attention

Xinmeng Xu, Yang Wang, Jie Jia, Binbin Chen, Dejun Li

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted by Interspeech 2022. arXiv admin note: substantial text overlap with arXiv:2101.06268

A Comprehensive Survey on Video Saliency Detection with Auditory Information: the Audio-visual Consistency Perceptual is the Key!

Chenglizhao Chen, Mengke Song, Wenfeng Song, Li Guo, Muwei Jian

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

PoCaP Corpus: A Multimodal Dataset for Smart Operating Room Speech Assistant using Interventional Radiology Workflow Analysis

Kubilay Can Demir, Matthias May, Axel Schmid, Michael Uder, Katharina Breininger, Tobias Weise, Andreas Maier, Seung Hee Yang

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI、eess.AS

Comments 8 pages, 4 figures, Text, Speech and Dialogue 2022 Conference

Towards Generalisable Audio Representations for Audio-Visual Navigation

Shunqi Mao, Chaoyi Zhang, Heng Wang, Weidong Cai

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments CVPR 2022 Embodied AI Workshop

Merkel Podcast Corpus: A Multimodal Dataset Compiled from 16 Years of Angela Merkel's Weekly Video Podcasts

Debjoy Saha, Shravan Nayak, Timo Baumann

专题命中 音频语音多模态 ：multimodal(title)；audio-visual(abstract)；分类 cs.CL、eess.AS

Comments Accepted at LREC 2022

A Closer Look at Audio-Visual Multi-Person Speech Recognition and Active Speaker Selection

Otavio Braga, Olivier Siohan

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments arXiv admin note: text overlap with arXiv:2205.05586

Cross-modal Contrastive Learning for Speech Translation

Rong Ye, Mingxuan Wang, Lei Li

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 cs.CL、eess.AS

Comments NAACL 2022 main conference (Long Paper)

Music Interpretation Analysis. A Multimodal Approach To Score-Informed Resynthesis of Piano Recordings

Federico Simonetta

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM、eess.AS

Comments PhD Thesis. Author: F. Simonetta; tutor: S. Ntalampiras; co-tutor: F. Avanzini; Università degli studi di Milano - Dipartimento di Informatica "Giovanni Degli Antoni", 2022 Apr 22

Audio-Visual Collaborative Representation Learning for Dynamic Saliency Prediction

Hailong Ning, Bin Zhao, Zhanxuan Hu, Lang He, Ercheng Pei

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM

MUGEN: A Playground for Video-Audio-Text Multimodal Understanding and GENeration

Thomas Hayes, Songyang Zhang, Xi Yin, Guan Pang, Sasha Sheng, Harry Yang, Songwei Ge, Qiyuan Hu, Devi Parikh

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Attention-Based Lip Audio-Visual Synthesis for Talking Face Generation in the Wild

Ganglai Wang, Peng Zhang, Lei Xie, Wei Huang, Yufei Zha

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM

Multilingual and Multimodal Abuse Detection

Rini Sharon, Heet Shah, Debdoot Mukherjee, Vikram Gupta

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments Submitted to Interspeech 2022

End-to-end multi-talker audio-visual ASR using an active speaker attention module

Richard Rose, Olivier Siohan

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CL、eess.AS

Comments 5 pages, 3 figures, 3 tables, 28 citations

Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis

Karren Yang, Dejan Markovic, Steven Krenn, Vasu Agrawal, Alexander Richard

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Enhancing Speaking Styles in Conversational Text-to-Speech Synthesis with Graph-based Multi-modal Context Modeling

Jingbei Li, Yi Meng, Chenyi Li, Zhiyong Wu, Helen Meng, Chao Weng, Dan Su

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted by ICASSP 2022

Balanced Multimodal Learning via On-the-fly Gradient Modulation

Xiaokang Peng, Yake Wei, Andong Deng, Dong Wang, Di Hu

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR 2022 (ORAL)

Improve few-shot voice cloning using multi-modal learning

Haitong Zhang, Yue Lin

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、eess.AS

Comments 2022 IEEE International Conference on Acoustics, Speech and Signal Processing

M2Lens: Visualizing and Explaining Multimodal Models for Sentiment Analysis

Xingbo Wang, Jianben He, Zhihua Jin, Muqiao Yang, Yong Wang, Huamin Qu

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.MM

Comments 11 pages, 7 figures. This paper is accepted by IEEE VIS, 2021. To appear in IEEE Transactions on Visualization and Computer Graphics (TVCG)

Journal ref IEEE Transactions on Visualization and Computer Graphics 28, no. 1 (2021): 802-812

An Empirical Study of Visual Features for DNN based Audio-Visual Speech Enhancement in Multi-talker Environments

Shrishti Saha Shetu, Soumitro Chakrabarty, Emanuël A. P. Habets

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Multimodal Depression Classification Using Articulatory Coordination Features And Hierarchical Attention Based Text Embeddings

Nadee Seneviratne, Carol Espy-Wilson

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted to ICASSP 2022. arXiv admin note: text overlap with arXiv:2104.04195

The impact of removing head movements on audio-visual speech enhancement

Zhiqi Kang, Mostafa Sadeghi, Radu Horaud, Xavier Alameda-Pineda, Jacob Donley, Anurag Kumar

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Unified Multimodal Punctuation Restoration Framework for Mixed-Modality Corpus

Yaoming Zhu, Liwei Wu, Shanbo Cheng, Mingxuan Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 5 pages, accepted by ICASSP'2022