多模态大模型

AlignVSR: Audio-Visual Cross-Modal Alignment for Visual Speech Recognition

Zehua Liu, Xiaolou Li, Chen Chen, Li Guo, Lantian Li, Dong Wang

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Separate in the Speech Chain: Cross-Modal Conditional Audio-Visual Target Speech Extraction

Zhaoxi Mu, Xinyu Yang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(title)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by IJCAI 2024

Improving Audio-Visual Speech Recognition by Lip-Subword Correlation Based Visual Pre-training and Cross-Modal Fusion Encoder

Yusheng Dai, Hang Chen, Jun Du, Xiaofei Ding, Ning Ding, Feijun Jiang, Chin-Hui Lee

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 6 pages, 2 figures, published in ICME2023

Multi-Modal Multi-Correlation Learning for Audio-Visual Speech Separation

Xiaoyu Wang, Xiangyu Kong, Xiulian Peng, Yan Lu

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 5 pages, accepted by interspeech2022

MPN: Multimodal Parallel Network for Audio-Visual Event Localization

Jiashuo Yu, Ying Cheng, Rui Feng

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments IEEE International Conference on Multimedia and Expo (ICME) 2021 Oral

Coordinated Joint Multimodal Embeddings for Generalized Audio-Visual Zeroshot Classification and Retrieval of Videos

Kranti Kumar Parida, Neeraj Matiyali, Tanaya Guha, Gaurav Sharma

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments To appear in WACV 2020, Project Page: https://cse.iitk.ac.in/users/kranti/avzsl.html

Qwen-MusicAVQA-7B: A Multimodal Model for Music Audio-Visual QA

Qwen-MusicAVQA-7B：一种用于音乐音频-视觉问答的多模态模型

Maryam Dehdashti

机构 * Inference Matter Labs（推理物质实验室） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；omni-modal(abstract)；分类 cs.CV、cs.MM

AI总结 该研究提出轻量级多模态模型Qwen-MusicAVQA-7B，通过连接冻结的Whisper编码器与Qwen2-VL-7B-Instruct，在MUSIC-AVQA等基准上实现高准确率，训练成本低，且发现音频时间信息保留程度影响下游问答准确率。

Comments 24 pages, 1 figure, 8 tables. Code: https://github.com/MKDehdashti/Qwen2-vl-audio Checkpoints: https://huggingface.co/MayaKD/qwen2-vl-audio

Beyond Generative Decoding: Discriminative Hidden-State Readout from a Native Omni-Modal LLM for Multimodal Sentiment Analysis

超越生成式解码：来自原生全模态大语言模型的判别性隐藏状态读出用于多模态情感分析

Bin Wen, Tien-Ping Tan

机构 * School of Computer Sciences, Universiti Sains Malaysia, Penang, Malaysia（计算机科学学院，马来西亚国际科学大学，槟城） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；omni-modal(title,abstract)；分类 cs.MM、eess.AS

AI总结 针对多模态情感分析中生成式读出将连续回归绑定到离散自回归解码导致精度和效率损失的问题，提出基于原生全模态大语言模型Qwen2.5-Omni-7B的Thinker模块的判别性读出方法，通过轻量回归头直接映射最终层隐藏状态，在单消费级GPU上实现最先进性能。

Comments 18 pages, 4 figures, 6 tables

Probing Cross-modal Information Hubs in Audio-Visual LLMs

探测音频-视觉大语言模型中的跨模态信息枢纽

Jihoo Jung, Chaeyoung Jung, Ji-Hoon Kim, Joon Son Chung

机构 * Department of Electrical Engineering, Korea Advanced Institute of Science ； The Graduate School of Advanced Imaging Science, Multimedia \& Film, Chung-Ang University, Seoul, Republic of Korea 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI、eess.AS

AI总结 本文研究了音频-视觉大语言模型中音频与视觉模态间的跨模态信息流动，发现信息主要存储在sink tokens中，并提出一种无需训练的hallucination缓解方法。

Comments Accepted by ICML 2026

Evaluating Hallucinations in Audio-Visual Multimodal LLMs with Spoken Queries under Diverse Acoustic Conditions

通过多样声学条件下的语音查询评估音频-视觉多模态大语言模型的幻觉

Hansol Park, Hoseong Ahn, Junwon Moon, Yejin Lee, Kyuhong Shim

机构 * Department of Intelligent Software, Sungkyunkwan University（智能软件系，成均馆大学） ； Department of Computer Science and Engineering, Sungkyunkwan University（计算机科学与工程系，成均馆大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title)；image-text(abstract)；分类 cs.AI、eess.AS

AI总结 研究探讨语音查询对多模态幻觉的影响，通过RePOPE-Spk基准测试发现语音查询导致错误率显著增加，提出改进方法以提升语音界面可靠性。

Comments Submitted to Interspeech2026

A Survey on Cross-Modal Interaction Between Music and Multimodal Data

多模态数据与音乐交叉交互的综述

Sifei Li, Mining Tan, Feier Shen, Minyan Luo, Zijiao Yin, Fan Tang, Weiming Dong, Changsheng Xu

机构 * MAIS, Institute of Automation, Chinese Academy of Sciences, Beijing, 100190, China. E-mail: S. Li ； School of Artificial Intelligence, University of Chinese Academy of Sciences, Beijing, 100190, China. E-mail: F. Shen ； Institute of Computing Technology, Chinese Academy of Sciences, Beijing, 100190, China. E-mail: . 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(title,abstract)；分类 cs.MM、eess.AS

AI总结 本文综述了音乐与多模态数据的交叉交互，探讨了音乐在多模态学习中的作用，并提出了未来研究的方向。

Comments 34 pages, 7 figures

Session-Level Spoken Language Assessment with a Multimodal Foundation Model via Multi-Target Learning

Hong-Yun Lin, Jhen-Ke Lin, Chung-Chun Wang, Hao-Chien Lu, Berlin Chen

机构 * Department of Computer Science（计算机科学系） ； Information Engineering, National Taiwan Normal University（信息工程，台湾正常大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；分类 cs.CL、cs.AI

Comments Copyright 2025 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works

Teacher-Guided Pseudo Supervision and Cross-Modal Alignment for Audio-Visual Video Parsing

Yaru Chen, Ruohao Guo, Liting Gao, Yang Xiang, Qingyu Luo, Zhenbo Li, Wenwu Wang

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM

Multi-source Multimodal Progressive Domain Adaption for Audio-Visual Deception Detection

Ronghao Lin, Sijie Mai, Ying Zeng, Qiaolin He, Aolin Xiong, Haifeng Hu

机构 * Sun Yat-Sen University（中山大学） ； Nanyang Technological University（南洋理工大学） ； South China Normal University（华南师范大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM

Comments Accepted at ACM MM 2025 SVC Workshop

EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning

Zhenghao Xing, Xiaowei Hu, Chi-Wing Fu, Wenhai Wang, Jifeng Dai, Pheng-Ann Heng

机构 * The Chinese University of Hong Kong（香港中文大学） ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） ； Tsinghua University（清华大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title)；cross-modal(abstract)；分类 cs.CV、eess.AS

Investigating Vulnerabilities and Defenses Against Audio-Visual Attacks: A Comprehensive Survey Emphasizing Multimodal Models

Jinming Wen, Xinyi Wu, Shuai Zhao, Yanhao Jia, Yuwen Li

机构 * Jilin University（吉林大学） ； Shanghai Jiao Tong University（上海交通大学） ； Nanyang Technological University（南洋理工大学） ； Northeastern University（东北大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI、cs.MM

Lightweight Joint Audio-Visual Deepfake Detection via Single-Stream Multi-Modal Learning Framework

Kuiyuan Zhang, Wenjie Pei, Rushi Lan, Yifang Guo, Zhongyun Hua

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI、eess.AS

Towards Source Attribution of Singing Voice Deepfake with Multimodal Foundation Models

Orchid Chetia Phukan, Girish, Mohd Mujtaba Akhtar, Swarup Ranjan Behera, Priyabrata Mallick, Pailla Balakrishna Reddy, Arun Balaji Buduru, Rajesh Sharma

机构 * IIIT-DelhiIndia（印度IIIT-Delhi） ； UPESIndia（印度UPES） ； V.B.S.P.UIndia（印度V.B.S.P.U） ； Independent ResearcherIndia（印度独立研究者） ； Reliance AIIndia（印度Reliance AI） ； University of TartuEstonia（爱沙尼亚塔尔图大学） ； Plaksha UniversityIndia（印度Plaksha大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted to INTERSPEECH 2025

The Multimodal Information Based Speech Processing (MISP) 2025 Challenge: Audio-Visual Diarization and Recognition

Ming Gao, Shilong Wu, Hang Chen, Jun Du, Chin-Hui Lee, Shinji Watanabe, Jingdong Chen, Siniscalchi Sabato Marco, Odette Scharenborg

机构 * University of Science and Technology of China（中国科学技术大学） ； Georgia Institute of Technology（佐治亚理工学院） ； Carnegie Mellon University（卡内基梅隆大学） ； Northwestern Polytechnical University（西北工业大学） ； University of Palermo（巴勒莫大学） ； Delft University of Technology（代尔夫特理工大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；multi-modal(abstract)；分类 cs.AI、eess.AS

Comments Accepted by Interspeech 2025. Camera-ready version

Multimodal Class-aware Semantic Enhancement Network for Audio-Visual Video Parsing

Pengcheng Zhao, Jinxing Zhou, Yang Zhao, Dan Guo, Yanxiang Chen

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by AAAI-2025

Learning Video Temporal Dynamics with Cross-Modal Attention for Robust Audio-Visual Speech Recognition

Sungnyun Kim, Kangwook Jang, Sangmin Bae, Hoirin Kim, Se-Young Yun

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CL、eess.AS

Comments Accepted at SLT 2024 Main Conference; Code is available at https://github.com/sungnyun/avsr-temporal-dynamics

A multi-modal approach for identifying schizophrenia using cross-modal attention

Gowtham Premananth, Yashish M. Siriwardena, Philip Resnik, Carol Espy-Wilson

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted to Annual International Conference of the IEEE Engineering in Medicine and Biology Society 2024

Multimodal Transformer Distillation for Audio-Visual Synchronization

Xuanjun Chen, Haibin Wu, Chung-Che Wang, Hung-yi Lee, Jyh-Shing Roger Jang

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted by ICASSP 2024

CoAVT: A Cognition-Inspired Unified Audio-Visual-Text Pre-Training Model for Multimodal Processing

Xianghu Yue, Xiaohai Tian, Lu Lu, Malu Zhang, Zhizheng Wu, Haizhou Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM、eess.AS

LiveChat: Video Comment Generation from Audio-Visual Multimodal Contexts

Julien Lalanne, Raphael Bournet, Yi Yu

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.AI

CM-PIE: Cross-modal perception for interactive-enhanced audio-visual video parsing

Yaru Chen, Ruohao Guo, Xubo Liu, Peipei Wu, Guangyao Li, Zhenbo Li, Wenwu Wang

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM

Comments 5 pages, 3 figures, 15 references

MMCosine: Multi-Modal Cosine Loss Towards Balanced Audio-Visual Fine-Grained Learning

Ruize Xu, Ruoxuan Feng, Shi-Xiong Zhang, Di Hu

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM、eess.AS

Cross-modal Audio-visual Co-learning for Text-independent Speaker Verification

Meng Liu, Kong Aik Lee, Longbiao Wang, Hanyi Zhang, Chang Zeng, Jianwu Dang

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Audio-Visual Activity Guided Cross-Modal Identity Association for Active Speaker Detection

Rahul Sharma, Shrikanth Narayanan

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM

Comments Under review at OJSP

Complete Cross-triplet Loss in Label Space for Audio-visual Cross-modal Retrieval

Donghuo Zeng, Yanan Wang, Jianming Wu, Kazushi Ikeda

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI、cs.MM

Comments 9 pages, 5 figures, 3 tables, accepted by IEEE ISM 2022