多模态大模型

MOSA: Music Motion with Semantic Annotation Dataset for Cross-Modal Music Processing

Yu-Fen Huang, Nikki Moran, Simon Coleman, Jon Kelly, Shun-Hwa Wei, Po-Yin Chen, Yun-Hsin Huang, Tsung-Ping Chen, Yu-Chia Kuo, Yu-Chi Wei, Chih-Hsuan Li, Da-Yu Huang, Hsuan-Kai Kao, Ting-Wei Lin, Li Su

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 cs.AI、eess.AS

Comments IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024. 14 pages, 7 figures. Dataset is available on: https://github.com/yufenhuang/MOSA-Music-mOtion-and-Semantic-Annotation-dataset/tree/main and https://zenodo.org/records/11393449

Rene: A Pre-trained Multi-modal Architecture for Auscultation of Respiratory Diseases

Pengfei Zhang, Zhihang Zheng, Shichen Zhang, Minghao Yang, Shaojun Tang

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.AI、eess.AS

Improving Audio Codec-based Zero-Shot Text-to-Speech Synthesis with Multi-Modal Context and Large Language Model

Jinlong Xue, Yayue Deng, Yicheng Han, Yingming Gao, Ya Li

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted by Interspeech 2024

Audio-Visual Talker Localization in Video for Spatial Sound Reproduction

Davide Berghi, Philip J. B. Jackson

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Advancing Weakly-Supervised Audio-Visual Video Parsing via Segment-wise Pseudo Labeling

Jinxing Zhou, Dan Guo, Yiran Zhong, Meng Wang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM

Comments IJCV 2024 Accepted. arXiv admin note: substantial text overlap with arXiv:2303.02344

ViSpeR: Multilingual Audio-Visual Speech Recognition

Sanath Narayan, Yasser Abdelaziz Dahou Djilali, Ankit Singh, Eustache Le Bihan, Hakim Hacid

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CL、cs.AI

Video2Music: Suitable Music Generation from Videos using an Affective Multimodal Transformer model

Jaeyong Kang, Soujanya Poria, Dorien Herremans

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI、eess.AS

Journal ref Expert Systems with Applications 249 (2024): 123640

Hearing Touch: Audio-Visual Pretraining for Contact-Rich Manipulation

Jared Mejia, Victoria Dean, Tess Hellebrekers, Abhinav Gupta

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted to ICRA 2024

Reka Core, Flash, and Edge: A Series of Powerful Multimodal Language Models

Reka Team, Aitor Ormazabal, Che Zheng, Cyprien de Masson d'Autume, Dani Yogatama, Deyu Fu, Donovan Ong, Eric Chen, Eugenie Lamprecht, Hai Pham, Isaac Ong, Kaloyan Aleksiev, Lei Li, Matthew Henderson, Max Bain, Mikel Artetxe, Nishant Relan, Piotr Padlewski, Qi Liu, Ren Chen, Samuel Phua, Yazheng Yang, Yi Tay, Yuqi Wang, Zhongkai Zhu, Zhihui Xie

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Learning Spatial Features from Audio-Visual Correspondence in Egocentric Videos

Sagnik Majumder, Ziad Al-Halah, Kristen Grauman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to CVPR 2024

Fake it to make it: Using synthetic data to remedy the data shortage in joint multimodal speech-and-gesture synthesis

Shivam Mehta, Anna Deichler, Jim O'Regan, Birger Moëll, Jonas Beskow, Gustav Eje Henter, Simon Alexanderson

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、eess.AS

Comments 13+1 pages, 2 figures, accepted at the Human Motion Generation workshop (HuMoGen) at CVPR 2024

Conversational Speech Recognition by Learning Audio-textual Cross-modal Contextual Representation

Kun Wei, Bei Li, Hang Lv, Quan Lu, Ning Jiang, Lei Xie

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 cs.CL、eess.AS

Comments TASLP

Journal ref IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024

Audio-Visual Person Verification based on Recursive Fusion of Joint Cross-Attention

R. Gnana Praveen, Jahangir Alam

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to FG2024

ActiveRIR: Active Audio-Visual Exploration for Acoustic Environment Modeling

Arjun Somayazulu, Sagnik Majumder, Changan Chen, Kristen Grauman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Project page: https://vision.cs.utexas.edu/projects/active_rir/

Dynamic Cross Attention for Audio-Visual Person Verification

R. Gnana Praveen, Jahangir Alam

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to FG2024

Joint Multimodal Transformer for Emotion Recognition in the Wild

Paul Waligora, Haseeb Aslam, Osama Zeeshan, Soufiane Belharbi, Alessandro Lameiras Koerich, Marco Pedersoli, Simon Bacon, Eric Granger

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、eess.AS

Comments 10 pages, 4 figures, 6 tables, CVPRw 2024

Dynamic Modality and View Selection for Multimodal Emotion Recognition with Missing Modalities

Luciana Trinkaus Menon, Luiz Carlos Ribeiro Neduziak, Jean Paul Barddal, Alessandro Lameiras Koerich, Alceu de Souza Britto

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、eess.AS

Comments 15 pages

MLCA-AVSR: Multi-Layer Cross Attention Fusion based Audio-Visual Speech Recognition

He Wang, Pengcheng Guo, Pan Zhou, Lei Xie

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、eess.AS

Comments 5 pages, 3 figures Accepted at ICASSP 2024

M3TCM: Multi-modal Multi-task Context Model for Utterance Classification in Motivational Interviews

Sayed Muddashir Hossain, Jan Alexandersson, Philipp Müller

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted for publication at LREC-COLING'24

LastResort at SemEval-2024 Task 3: Exploring Multimodal Emotion Cause Pair Extraction as Sequence Labelling Task

Suyash Vardhan Mathur, Akshett Rai Jindal, Hardik Mittal, Manish Shrivastava

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Multi-Stage Multi-Modal Pre-Training for Automatic Speech Recognition

Yash Jain, David Chan, Pranav Dheram, Aparna Khare, Olabanji Shonibare, Venkatesh Ravichandran, Shalini Ghosh

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted in LREC-COLING 2024 - The 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation

Siamese Vision Transformers are Scalable Audio-visual Learners

Yan-Bo Lin, Gedas Bertasius

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

A Multimodal Approach to Device-Directed Speech Detection with Large Language Models

Dominik Wagner, Alexander Churchill, Siddharth Sigtia, Panayiotis Georgiou, Matt Mirsamadi, Aarshee Mishra, Erik Marchi

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments arXiv admin note: text overlap with arXiv:2312.03632

Unsupervised Audio-Visual Segmentation with Modality Alignment

Swapnil Bhosale, Haosen Yang, Diptesh Kanojia, Jiangkang Deng, Xiatian Zhu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.AI

RTFS-Net: Recurrent Time-Frequency Modelling for Efficient Audio-Visual Speech Separation

Samuel Pegg, Kai Li, Xiaolin Hu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted by The Twelfth International Conference on Learning Representations (ICLR) 2024, see https://openreview.net/forum?id=PEuDO2EiDr

Exploring Multi-Modal Control in Music-Driven Dance Generation

Ronghui Li, Yuqin Dai, Yachao Zhang, Jun Li, Jian Yang, Jie Guo, Xiu Li

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CV、eess.AS

Voting-based Multimodal Automatic Deception Detection

Lana Touma, Mohammad Al Horani, Manar Tailouni, Anas Dahabiah, Khloud Al Jallad

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Learning Co-Speech Gesture for Multimodal Aphasia Type Detection

Daeun Lee, Sejung Son, Hyolim Jeon, Seungbae Kim, Jinyoung Han

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2023 accepted

Journal ref EMNLP 2023:Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing

Multi-Modal Discussion Transformer: Integrating Text, Images and Graph Transformers to Detect Hate Speech on Social Media

Liam Hebert, Gaurav Sahu, Yuxuan Guo, Nanda Kishore Sreenivas, Lukasz Golab, Robin Cohen

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、cs.MM

Comments Accepted to AAAI 2024 (AI for Social Impact Track)

SynthScribe: Deep Multimodal Tools for Synthesizer Sound Retrieval and Exploration

Stephen Brade, Bryan Wang, Mauricio Sousa, Gregory Lee Newsome, Sageev Oore, Tovi Grossman

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI、eess.AS