多模态大模型

Language Model Mapping in Multimodal Music Learning: A Grand Challenge Proposal

Daniel Chin, Gus Xia

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

Global-Local Distillation Network-Based Audio-Visual Speaker Tracking with Incomplete Modalities

Yidi Li, Yihan Li, Yixin Guo, Bin Ren, Zhenhuan Xu, Hao Guo, Hong Liu, Nicu Sebe

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments We request to withdraw our paper from arXiv due to unresolved author disagreements about the data interpretation and study conclusions. To maintain scientific integrity, we believe withdrawing the paper is necessary. We regret any confusion caused

LSTMSE-Net: Long Short Term Speech Enhancement Network for Audio-visual Speech Enhancement

Arnav Jain, Jasmer Singh Sanjotra, Harshvardhan Choudhary, Krish Agrawal, Rupal Shah, Rohan Jha, M. Sajid, Amir Hussain, M. Tanveer

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.MM、eess.AS

Journal ref INTERSPEECH 2024

AVTrustBench: Assessing and Enhancing Reliability and Robustness in Audio-Visual LLMs

Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Yaoting Wang, Mohamed Elhoseiny, Ruohan Gao, Dinesh Manocha

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

AV-EmoDialog: Chat with Audio-Visual Users Leveraging Emotional Cues

Se Jin Park, Yeonju Kim, Hyeongseop Rha, Bella Godiva, Yong Man Ro

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Mozart's Touch: A Lightweight Multi-modal Music Generation Framework Based on Pre-Trained Large Models

Jiajun Li, Tianze Xu, Xuesong Chen, Xinrui Yao, Shuchang Liu

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

Comments 10 pages, 2 figures, submitted to AIGC 2024

Talking Head Generation Driven by Speech-Related Facial Action Units and Audio- Based on Multimodal Representation Fusion

Sen Chen, Zhilei Liu, Jiaxing Liu, Longbiao Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments arXiv admin note: text overlap with arXiv:2110.09951

Quality-Aware End-to-End Audio-Visual Neural Speaker Diarization

Mao-Kui He, Jun Du, Shu-Tong Niu, Qing-Feng Liu, Chin-Hui Lee

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.MM、eess.AS

RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction

Yuwei Zhang, Tong Xia, Aaqib Saeed, Cecilia Mascolo

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

Multimodal Speech Enhancement Using Burst Propagation

Mohsin Raza, Leandro A. Passos, Ahmed Khubaib, Ahsan Adeel

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.MM、eess.AS

ModalityMirror: Improving Audio Classification in Modality Heterogeneity Federated Learning with Multimodal Distillation

Tiantian Feng, Tuo Zhang, Salman Avestimehr, Shrikanth S. Narayanan

专题命中 音频语音多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.AI、eess.AS

Integrating Audio, Visual, and Semantic Information for Enhanced Multimodal Speaker Diarization

Luyao Cheng, Hui Wang, Siqi Zheng, Yafeng Chen, Rongjie Huang, Qinglin Zhang, Qian Chen, Xihao Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Unraveling Instance Associations: A Closer Look for Audio-Visual Segmentation

Yuanhong Chen, Yuyuan Liu, Hu Wang, Fengbei Liu, Chong Wang, Helen Frazer, Gustavo Carneiro

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Code is available at https://github.com/cyh-0/CAVP

Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection

Yunfeng Fan, Wenchao Xu, Haozhao Wang, Fushuo Huo, Jinyu Chen, Song Guo

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM

Comments Accepted by ECCV24, 23 pages

Cascaded Cross-Modal Transformer for Audio-Textual Classification

Nicolae-Catalin Ristea, Andrei Anghel, Radu Tudor Ionescu

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL、eess.AS

Comments Accepted for publication in Artificial Intelligence Review

Ref-AVS: Refer and Segment Objects in Audio-Visual Scenes

Yaoting Wang, Peiwen Sun, Dongzhan Zhou, Guangyao Li, Honggang Zhang, Di Hu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ECCV2024

A Joint Cross-Attention Model for Audio-Visual Fusion in Dimensional Emotion Recognition

R. Gnana Praveen, Wheidima Carneiro de Melo, Nasib Ullah, Haseeb Aslam, Osama Zeeshan, Théo Denorme, Marco Pedersoli, Alessandro Koerich, Simon Bacon, Patrick Cardinal, Eric Granger

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、eess.AS

Comments arXiv admin note: text overlap with arXiv:2111.05222

Cross Attentional Audio-Visual Fusion for Dimensional Emotion Recognition

R. Gnana Praveen, Eric Granger, Patrick Cardinal

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、eess.AS

Comments Accepted in FG2021

MSRS: Training Multimodal Speech Recognition Models from Scratch with Sparse Mask Optimization

Adriana Fernandez-Lopez, Honglie Chen, Pingchuan Ma, Lu Yin, Qiao Xiao, Stavros Petridis, Shiwei Liu, Maja Pantic

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM

Comments Accepted at Interspeech 2024

Discrete Multimodal Transformers with a Pretrained Large Language Model for Mixed-Supervision Speech Processing

Viet Anh Trinh, Rosy Southwell, Yiwen Guan, Xinlu He, Zhiyong Wang, Jacob Whitehill

专题命中 音频语音多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CL、eess.AS

The MuSe 2024 Multimodal Sentiment Analysis Challenge: Social Perception and Humor Recognition

Shahin Amiriparian, Lukas Christ, Alexander Kathan, Maurice Gerczuk, Niklas Müller, Steffen Klug, Lukas Stappen, Andreas König, Erik Cambria, Björn Schuller, Simone Eulitz

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CL、cs.AI

Target Speech Diarization with Multimodal Prompts

Yidi Jiang, Ruijie Tao, Zhengyang Chen, Yanmin Qian, Haizhou Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.MM、eess.AS

Comments 13 pages, 7 figures

What do MLLMs hear? Examining reasoning with text and sound components in Multimodal Large Language Models

Enis Berk Çoban, Michael I. Mandel, Johanna Devaney

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、eess.AS

Comments 9 pages

Explicit Correlation Learning for Generalizable Cross-Modal Deepfake Detection

Cai Yu, Shan Jia, Xiaomeng Fu, Jin Liu, Jiahe Tian, Jiao Dai, Xi Wang, Siwei Lyu, Jizhong Han

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

Comments accepted by ICME 2024

Semantically consistent Video-to-Audio Generation using Multimodal Language Large Model

Gehui Chen, Guan'an Wang, Xiaowen Huang, Jitao Sang

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.MM、eess.AS

Leveraging Speech for Gesture Detection in Multimodal Communication

Esam Ghaleb, Ilya Burenko, Marlou Rasenberg, Wim Pouw, Ivan Toni, Peter Uhrig, Anna Wilson, Judith Holler, Aslı Özyürek, Raquel Fernández

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

A Cross-Modal Approach to Silent Speech with LLM-Enhanced Recognition

Tyler Benster, Guy Wilson, Reshef Elisha, Francis R Willett, Shaul Druckmann

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.AI、eess.AS

Evaluation of Audio-Visual Alignments in Visually Grounded Speech Models

Khazar Khorrami, Okko Räsänen

专题命中 音频语音多模态 ：audio-visual(title)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments To be published in Proc. Interspeech-2021, Brno, Czech Republic

A multimodal dynamical variational autoencoder for audiovisual speech representation learning

Samir Sadok, Simon Leglaive, Laurent Girin, Xavier Alameda-Pineda, Renaud Séguier

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.MM、eess.AS

Comments 14 figures, https://samsad35.github.io/site-mdvae/

Computation and Parameter Efficient Multi-Modal Fusion Transformer for Cued Speech Recognition

Lei Liu, Li Liu, Haizhou Li

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、eess.AS

Comments Accepted by TASLP