多模态大模型

A multimodal LLM for the non-invasive decoding of spoken text from brain recordings

Youssef Hmamouche, Ismail Chihab, Lahoucine Kdouri, Amal El Fallah Seghrouchni

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments 15 pages, 4 figures

EMMeTT: Efficient Multimodal Machine Translation Training

Piotr Żelasko, Zhehuai Chen, Mengru Wang, Daniel Galvez, Oleksii Hrinchuk, Shuoyang Ding, Ke Hu, Jagadeesh Balam, Vitaly Lavrukhin, Boris Ginsburg

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments 4 pages, submitted to ICASSP 2025

M$^{3}$V: A multi-modal multi-view approach for Device-Directed Speech Detection

Anna Wang, Da Liu, Zhiyu Zhang, Shengqiang Liu, Jie Gao, Yali Li

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.MM、eess.AS

Rhythmic Foley: A Framework For Seamless Audio-Visual Alignment In Video-to-Audio Synthesis

Zhiqi Huang, Dan Luo, Jun Wang, Huan Liao, Zhiheng Li, Zhiyong Wu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Stepping Stones: A Progressive Training Strategy for Audio-Visual Semantic Segmentation

Juncheng Ma, Peiwen Sun, Yaoting Wang, Di Hu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.AI

Comments ECCV2024 poster. Project url: https://gewu-lab.github.io/stepping_stones

Sim2Real Transfer for Audio-Visual Navigation with Frequency-Adaptive Acoustic Field Prediction

Changan Chen, Jordi Ramos, Anshul Tomar, Kristen Grauman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、eess.AS

Comments Camera ready version for IROS 2024. Project page: https://vision.cs.utexas.edu/projects/sim2real/

Audio-Visual Speaker Diarization: Current Databases, Approaches and Challenges

Victoria Mingote, Alfonso Ortega, Antonio Miguel, Eduardo Lleida

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Segment Beyond View: Handling Partially Missing Modality for Audio-Visual Semantic Segmentation

Renjie Wu, Hu Wang, Feras Dayoub, Hsiang-Ting Chen

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments AAAI-24 (Fixed some erros)

Multi-Modal Multi-Granularity Tokenizer for Chu Bamboo Slip Scripts

Yingfa Chen, Chenlong Hu, Cong Feng, Chenyang Song, Shi Yu, Xu Han, Zhiyuan Liu, Maosong Sun

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments 12 pages, 3 figures

Efficient Long-distance Latent Relation-aware Graph Neural Network for Multi-modal Emotion Recognition in Conversations

Yuntao Shou, Wei Ai, Jiayi Du, Tao Meng, Haiyan Liu, Nan Yin

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Comments 11 pages, 3 tables

Attend-Fusion: Efficient Audio-Visual Fusion for Video Classification

Mahrukh Awan, Asmar Nadeem, Muhammad Junaid Awan, Armin Mustafa, Syed Sameed Husain

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.AI

XLAVS-R: Cross-Lingual Audio-Visual Speech Representation Learning for Noise-Robust Speech Perception

HyoJung Han, Mohamed Anwar, Juan Pino, Wei-Ning Hsu, Marine Carpuat, Bowen Shi, Changhan Wang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CL、eess.AS

Comments ACL2024

Large AI Model Empowered Multimodal Semantic Communications

Feibo Jiang, Li Dong, Yubo Peng, Kezhi Wang, Kun Yang, Cunhua Pan, Xiaohu You

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Accepted by IEEE CM

Open-Vocabulary Audio-Visual Semantic Segmentation

Ruohao Guo, Liao Qu, Dantong Niu, Yanyu Qi, Wenzhen Yue, Ji Shi, Bowei Xing, Xianghua Ying

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、cs.MM

Comments Accepted by ACM MM 2024 (Oral)

RAVSS: Robust Audio-Visual Speech Separation in Multi-Speaker Scenarios with Missing Visual Cues

Tianrui Pan, Jie Liu, Bohan Wang, Jie Tang, Gangshan Wu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted by MM 2024

UNQA: Unified No-Reference Quality Assessment for Audio, Image, Video, and Audio-Visual Content

Yuqin Cao, Xiongkuo Min, Yixuan Gao, Wei Sun, Weisi Lin, Guangtao Zhai

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Multimodal Input Aids a Bayesian Model of Phonetic Learning

Sophia Zhi, Roger P. Levy, Stephan C. Meylan

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments 12 pages, 5 figures

Putting GPT-4o to the Sword: A Comprehensive Evaluation of Language, Vision, Speech, and Multimodal Proficiency

Sakib Shahriar, Brady Lund, Nishith Reddy Mannuru, Muhammad Arbab Arshad, Kadhim Hayawi, Ravi Varma Kumar Bevara, Aashrith Mannuru, Laiba Batool

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

AVCap: Leveraging Audio-Visual Features as Text Tokens for Captioning

Jongsuk Kim, Jiwon Shin, Junmo Kim

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CL、eess.AS

Comments Interspeech 2024

A Framework for Multimodal Medical Image Interaction

Laura Schütz, Sasan Matinfar, Gideon Schafroth, Navid Navab, Merle Fairhurst, Arthur Wagner, Benedikt Wiestler, Ulrich Eck, Nassir Navab

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted for publication in IEEE TVCG; presentation at IEEE ISMAR 2024

Towards Multimodal Sentiment Analysis Debiasing via Bias Purification

Dingkang Yang, Mingcheng Li, Dongling Xiao, Yang Liu, Kun Yang, Zhaoyu Chen, Yuzheng Wang, Peng Zhai, Ke Li, Lihua Zhang

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by ECCV 2024

A Toolchain for Comprehensive Audio/Video Analysis Using Deep Learning Based Multimodal Approach (A use case of riot or violent context detection)

Lam Pham, Phat Lam, Tin Nguyen, Hieu Tang, Alexander Schindler

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI、eess.AS

S3: A Simple Strong Sample-effective Multimodal Dialog System

Elisei Rykov, Egor Malkershin, Alexander Panchenko

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Towards Probing Speech-Specific Risks in Large Multimodal Models: A Taxonomy, Benchmark, and Insights

Hao Yang, Lizhen Qu, Ehsan Shareghi, Gholamreza Haffari

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

CoLM-DSR: Leveraging Neural Codec Language Modeling for Multi-Modal Dysarthric Speech Reconstruction

Xueyuan Chen, Dongchao Yang, Dingdong Wang, Xixin Wu, Zhiyong Wu, Helen Meng

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted by Interspeech 2024

EquiAV: Leveraging Equivariance for Audio-Visual Contrastive Learning

Jongsuk Kim, Hyeongkeun Lee, Kyeongha Rho, Junmo Kim, Joon Son Chung

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、cs.MM

Comments 15 pages, 3 figures; Accepted to ICML 2024 (camera ready version)

Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications

Paul Pu Liang, Chun Kai Ling, Yun Cheng, Alex Obolenskiy, Yudong Liu, Rohan Pandey, Alex Wilf, Louis-Philippe Morency, Ruslan Salakhutdinov

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments ICLR 2024, Code available at: https://github.com/pliang279/PID

Zero-Shot Fake Video Detection by Audio-Visual Consistency

Xiaolou Li, Zehua Liu, Chen Chen, Lantian Li, Li Guo, Dong Wang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments to be published in INTERSPEECH 2024

Labeling Comic Mischief Content in Online Videos with a Multimodal Hierarchical-Cross-Attention Model

Elaheh Baharlouei, Mahsa Shafaei, Yigeng Zhang, Hugo Jair Escalante, Thamar Solorio

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

MM-KWS: Multi-modal Prompts for Multilingual User-defined Keyword Spotting

Zhiqi Ai, Zhiyong Chen, Shugong Xu

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted at INTERSPEECH 2024