多模态大模型

A Pre-trained Audio-Visual Transformer for Emotion Recognition

Minh Tran, Mohammad Soleymani

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by IEEE ICASSP 2022

Fast End-to-End Speech Recognition via Non-Autoregressive Models and Cross-Modal Knowledge Transferring from BERT

Ye Bai, Jiangyan Yi, Jianhua Tao, Zhengkun Tian, Zhengqi Wen, Shuai Zhang

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 14 pages, 7 figures

Improving Multi-Modal Learning with Uni-Modal Teachers

Chenzhuang Du, Tingle Li, Yichen Liu, Zixin Wen, Tianyu Hua, Yue Wang, Hang Zhao

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(abstract)

Highlight Timestamp Detection Model for Comedy Videos via Multimodal Sentiment Analysis

Fan Huang

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Pose-Controllable Talking Face Generation by Implicitly Modularized Audio-Visual Representation

Hang Zhou, Yasheng Sun, Wayne Wu, Chen Change Loy, Xiaogang Wang, Ziwei Liu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021. Code and models are available at https://github.com/Hangz-nju-cuhk/Talking-Face_PC-AVS

The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes

Douwe Kiela, Hamed Firooz, Aravind Mohan, Vedanuj Goswami, Amanpreet Singh, Pratik Ringshia, Davide Testuggine

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2020

Positive Sample Propagation along the Audio-Visual Event Line

Jinxing Zhou, Liang Zheng, Yiran Zhong, Shijie Hao, Meng Wang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to CVPR 2021. Code is available at https://github.com/jasongief/PSP_CVPR_2021

Detecting Hate Speech in Memes Using Multimodal Deep Learning Approaches: Prize-winning solution to Hateful Memes Challenge

Riza Velioglu, Jewgeni Rose

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Presented at NeurIPS (2020)

Jointly Optimizing Sensing Pipelines for Multimodal Mixed Reality Interaction

Darshana Rathnayake, Ashen de Silva, Dasun Puwakdandawa, Lakmal Meegahapola, Archan Misra, Indika Perera

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)

Comments 17th IEEE International Conference on Mobile Ad-Hoc and Sensor Systems (MASS) - 2020

MAST: Multimodal Abstractive Summarization with Trimodal Hierarchical Attention

Aman Khullar, Udit Arora

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments To appear in the first EMNLP Workshop on NLP Beyond Text, 2020. Aman Khullar and Udit Arora have equal contribution

Unsupervised vs. transfer learning for multimodal one-shot matching of speech and images

Leanne Nortje, Herman Kamper

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Accepted at Interspeech 2020

Strong and Simple Baselines for Multimodal Utterance Embeddings

Paul Pu Liang, Yao Chong Lim, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Louis-Philippe Morency

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments NAACL 2019 oral presentation

deepsing: Generating Sentiment-aware Visual Stories using Cross-modal Music Translation

Nikolaos Passalis, Stavros Doropoulos

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Advances in Online Audio-Visual Meeting Transcription

Takuya Yoshioka, Igor Abramovski, Cem Aksoylar, Zhuo Chen, Moshe David, Dimitrios Dimitriadis, Yifan Gong, Ilya Gurvich, Xuedong Huang, Yan Huang, Aviv Hurvitz, Li Jiang, Sharon Koubi, Eyal Krupka, Ido Leichter, Changliang Liu, Partha Parthasarathy, Alon Vinnikov, Lingfeng Wu, Xiong Xiao, Wayne Xiong, Huaming Wang, Zhenghao Wang, Jun Zhang, Yong Zhao, Tianyan Zhou

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments To appear in Proc. IEEE ASRU Workshop 2019

Recurrent Neural Network Transducer for Audio-Visual Speech Recognition

Takaki Makino, Hank Liao, Yannis Assael, Brendan Shillingford, Basilio Garcia, Otavio Braga, Olivier Siohan

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Will be presented in 2019 IEEE Automatic Speech Recognition and Understanding Workshop (ASRU 2019)

DurIAN: Duration Informed Attention Network For Multimodal Synthesis

Chengzhu Yu, Heng Lu, Na Hu, Meng Yu, Chao Weng, Kun Xu, Peng Liu, Deyi Tuo, Shiyin Kang, Guangzhi Lei, Dan Su, Dong Yu

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Towards Generating Ambisonics Using Audio-Visual Cue for Virtual Reality

Aakanksha Rana, Cagri Ozcinar, Aljoscha Smolic

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)

Speaker-Targeted Audio-Visual Models for Speech Recognition in Cocktail-Party Environments

Guan-Lin Chao, William Chan, Ian Lane

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Published in INTERSPEECH 2016

AVA-ActiveSpeaker: An Audio-Visual Dataset for Active Speaker Detection

Joseph Roth, Sourish Chaudhuri, Ondrej Klejch, Radhika Marvin, Andrew Gallagher, Liat Kaver, Sharadh Ramaswamy, Arkadiusz Stopczynski, Cordelia Schmid, Zhonghua Xi, Caroline Pantofaru

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Deep Multimodal Clustering for Unsupervised Audiovisual Learning

Di Hu, Feiping Nie, Xuelong Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by CVPR2019

End-to-End Audio Visual Scene-Aware Dialog using Multimodal Attention-Based Video Features

Chiori Hori, Huda Alamri, Jue Wang, Gordon Wichern, Takaaki Hori, Anoop Cherian, Tim K. Marks, Vincent Cartillier, Raphael Gontijo Lopes, Abhishek Das, Irfan Essa, Dhruv Batra, Devi Parikh

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments A prototype system for the Audio Visual Scene-aware Dialog (AVSD) at DSTC7

A Deep Learning Approach for Multimodal Deception Detection

Gangeshwar Krishnamurthy, Navonil Majumder, Soujanya Poria, Erik Cambria

专题命中 音频语音多模态 ：multimodal(title)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at the 19th International Conference on Computational Linguistics and Intelligent Text Processing (CICLing), 2018

AUREXA-SE: Audio-Visual Unified Representation Exchange Architecture with Cross-Attention and Squeezeformer for Speech Enhancement

M. Sajid, Deepanshu Gupta, Yash Modi, Sanskriti Jain, Harshith Jai Surya Ganji, A. Rahaman, Harshvardhan Choudhary, Nasir Saleem, Amir Hussain, M. Tanveer

机构 * Indian Institute of Technology Indore（印度理工学院印第安纳分校） ； School of Computing, Edinburgh Napier University（爱丁堡纳皮尔大学计算机学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、cs.MM

Journal ref INTERSPEECH 2025 - 4th COG-MHEAR Workshop on Audio-Visual Speech Enhancement (AVSEC)

A Multi-Task, Multi-Modal Approach for Predicting Categorical and Dimensional Emotions

Alex-Răzvan Ispas, Théo Deschamps-Berger, Laurence Devillers

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI；multimodal(comments)

Comments Companion Publication of the 25th International Conference on Multimodal Interaction (pp. 311-317)

Multimodal Approach for Assessing Neuromotor Coordination in Schizophrenia Using Convolutional Neural Networks

Yashish M. Siriwardena, Chris Kitchen, Deanna L. Kelly, Carol Espy-Wilson

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM、eess.AS

Comments 5 pages. arXiv admin note: text overlap with arXiv:2102.07054

Journal ref Proceedings of the 2021 International Conference on Multimodal Interaction

Learning Audio-Visual Dereverberation

Changan Chen, Wei Sun, David Harwath, Kristen Grauman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted at ICASSP 2023. This is the longer version of the five-page camera-ready paper. Project page: https://vision.cs.utexas.edu/projects/learning-audio-visual-dereverberation

Shapes of Emotions: Multimodal Emotion Recognition in Conversations via Emotion Shifts

Harsh Agarwal, Keshav Bansal, Abhinav Joshi, Ashutosh Modi

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 13 pages, Accepted at Workshop on Performance and Interpretability Evaluations of Multimodal, Multipurpose, Massive-Scale Models, COLING 2022

Sound Adversarial Audio-Visual Navigation

Yinfeng Yu, Wenbing Huang, Fuchun Sun, Changan Chen, Yikai Wang, Xiaohong Liu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments This work aims to do an adversarial sound intervention for robust audio-visual navigation

Multimodal Emotion Recognition with High-level Speech and Text Features

Mariana Rodrigues Makiuchi, Kuniaki Uto, Koichi Shinoda

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted at ASRU 2021. Code available at https://github.com/mmakiuchi/multimodal_emotion_recognition

Dyadic Speech-based Affect Recognition using DAMI-P2C Parent-child Multimodal Interaction Dataset

Huili Chen, Yue Zhang, Felix Weninger, Rosalind Picard, Cynthia Breazeal, Hae Won Park

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted by the 2020 International Conference on Multimodal Interaction (ICMI'20)