多模态大模型

Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks

Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Yu Tsao, Hsiu-Wen Chang, Hsin-Min Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments This paper is the same as arXiv:1703.10893v6. Apologies for the inconvenience. arXiv admin note: text overlap with arXiv:1703.10893

Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdelrahman Mohamed

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments ICLR 2022

Cross-Modal learning for Audio-Visual Video Parsing

Jatin Lamba, Abhishek, Jayaprakash Akula, Rishabh Dabral, Preethi Jyothi, Ganesh Ramakrishnan

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Work accepted at Interspeech 2021

Looking into Your Speech: Learning Cross-modal Affinity for Audio-visual Speech Separation

Jiyoung Lee, Soo-Whan Chung, Sunok Kim, Hong-Goo Kang, Kwanghoon Sohn

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments CVPR 2021. The first two authors contributed equally to this work. Project page: https://caffnet.github.io

Can audio-visual integration strengthen robustness under multimodal attacks?

Yapeng Tian, Chenliang Xu

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments CVPR 2021

Learning Audio-Visual Correlations from Variational Cross-Modal Generation

Ye Zhu, Yu Wu, Hugo Latapie, Yi Yang, Yan Yan

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to ICASSP 2021

AVGZSLNet: Audio-Visual Generalized Zero-Shot Learning by Reconstructing Label Features from Multi-Modal Embeddings

Pratik Mazumder, Pravendra Singh, Kranti Kumar Parida, Vinay P. Namboodiri

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title)；cross-modal(abstract)；分类 cs.CV、eess.AS

Comments Accepted in WACV 2021

Large Language Models are Strong Audio-Visual Speech Recognition Learners

Umberto Cappellazzo, Minsu Kim, Honglie Chen, Pingchuan Ma, Stavros Petridis, Daniele Falavigna, Alessio Brutti, Maja Pantic

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；multi-modal(abstract)；MLLM(abstract)

Comments Accepted for publication at ICASSP 2025. The code and checkpoints are available here: https://github.com/umbertocappellazzo/Llama-AVSR

Test-Time Scaling in Multimodal Foundation Models: A Comprehensive Survey of Generation and Reasoning

多模态基础模型中的测试时扩展：生成与推理的综合调查

Cong Wan, Ying He, Zhongzhan Huang, Hefeng Wu

机构 * Sun Yat-sen University（中山大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；分类 cs.CV

AI总结 本文首次系统综述多模态基础模型中的测试时扩展（TTS）方法，提出统一分类框架（采样、反馈、搜索三类），总结应用与基准，并讨论未来方向。

Comments Accepted by ACL 2026, Findings

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

先分后融：通过模态特定的推理链减轻音频视觉大语言模型中的跨模态干扰

Xuanchen Li, Yuheng Lu, Chenrui Cui, Tianrui Wang, Zikang Huang, Yu Jiang, Long Zhou, Longbiao Wang, Jianwu Dang

机构 * Tianjin Key Laboratory of Cognitive Computing（天津认知计算实验室） ； Tianjin University（天津大学） ； Huiyan Technology Company, Ltd.（慧颜科技有限公司） ； Chinese Academy of Sciences（中国科学院） ； Tencent（腾讯） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI

AI总结 本文提出SFFL框架，通过模态特定的推理链减少跨模态干扰，提升音频视觉问答的准确性和鲁棒性，实验显示在通用AVQA基准和跨模态幻觉基准上分别提升5.16%和11.17%。

MAViD: A Multimodal Framework for Audio-Visual Dialogue Understanding and Generation

MAViD：一种多模态框架用于音频-视觉对话理解和生成

Youxin Pang, Jiajun Liu, Lingfeng Tan, Yong Zhang, Feng Gao, Xiang Deng, Zhuoliang Kang, Xiaoming Wei, Yebin Liu

机构 * Tsinghua University（清华大学） ； Meituan（美团） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

AI总结 MAViD提出了一种多模态框架，通过Conductor-Creator架构实现音频-视觉对话的理解与生成，结合自回归和扩散模型提升长时多模态内容生成质量。

CAE-AV: Improving Audio-Visual Learning via Cross-modal Interactive Enrichment

CAE-AV：通过跨模态交互增强改进音频-视觉学习

Yunzuo Hu, Wen Li, Jing Zhang

机构 * School of Information Science and Engineering, East China University of Science and Technology (ECUST)（信息科学与工程学院，东华大学） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

AI总结 CAE-AV通过跨模态交互增强框架，解决音频-视觉学习中的模态不对齐问题，提升表示质量和任务性能。

Comments 13 pages, 8 figures

Cross-Modal Bottleneck Fusion For Noise Robust Audio-Visual Speech Recognition

跨模态瓶颈融合用于噪声鲁棒的音频视觉语音识别

Seaone Ok, Min Jun Choi, Eungbeom Kim, Seungu Han, Kyogu Lee

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

AI总结 CoBRA通过跨模态瓶颈融合机制，在噪声环境下提升音频视觉语音识别的鲁棒性和效率。

Comments 5 pages, 3 figures, ICASSP 2026 Accepted

From Contrast to Commonality: Audio Commonality Captioning for Enhanced Audio-Text Cross-modal Understanding in Multimodal LLMs

从对比到共同性：用于增强多模态大语言模型中音频-文本跨模态理解的音频共同性描述

Yuhang Jia, Xu Zhang, Yujie Guo, Yang Chen, Shiwan Zhao

机构 * TMCC, College of Computer Science, Nankai University, Tianjin, China（TMCC，计算机科学学院，南开大学，天津，中国） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(title,abstract)；分类 eess.AS

AI总结 本文提出音频共同性描述方法，旨在增强多模态大语言模型中音频与文本的跨模态理解，通过捕捉音频片段间的共享语义以改善模型的泛化能力与任务特定性能。

MAVERIX: Multimodal Audio-Visual Evaluation and Recognition IndeX

MAVERIX：多模态音频视觉评估与识别指数

Liuyue Xie, Avik Kuthiala, George Z. Wei, Ce Zheng, Ananya Bal, Mosam Dabhi, Liting Wen, Taru Rustagi, Ethan Lai, Sushil Khyalia, Rohan Choudhury, Morteza Ziyadi, Xu Zhang, Hao Yang, László A. Jeni

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title)；分类 cs.CV、cs.AI、cs.MM

AI总结 MAVERIX是一个用于评估多模态模型音频视觉整合能力的统一基准，通过精心设计的问题展示模型在跨模态理解上的性能，揭示了与人类水平的显著差距。

Journal ref AAAI 2026

ERF-BA-TFD+: A Multimodal Model for Audio-Visual Deepfake Detection

ERF-BA-TFD+: 一种用于音频视觉深度伪造检测的多模态模型

Xin Zhang, Jiaming Chu, Jian Zhao, Yuchu Jiang, Xu Yang, Lei Jin, Chi Zhang, Xuelong Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.AI

AI总结 ERF-BA-TFD+通过结合增强接收场和音频视觉融合，提出了一种多模态深度伪造检测模型，在DDL-AV数据集上实现了最先进的检测性能。

Comments The paper is withdrawn after discovering a flaw in the theoretical derivation presented in Section Method. The incorrect step leads to conclusions that are not supported by the corrected derivation. We plan to reconstruct the argument and will release an updated version once the issue is fully resolved

Surprisingly Fragile: Assessing and Addressing Prompt Instability in Multimodal Foundation Models

Ian Stewart, Sameera Horawalavithana, Brendan Kennedy, Sai Munikoti, Karl Pazdernik

机构 * Pacific Northwest National Laboratory（太平洋西北国家实验室） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；分类 cs.CL

Comments arxiv

Intelligent Carrier Allocation: A Cross-Modal Reasoning Framework for Adaptive Multimodal Steganography

Abhirup Das, Pranav Dudani, Shruti Sharma, Ravi Kumar C.

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(title,abstract)；分类 cs.MM

Comments 8 pages, 10 figures

Fork-Merge Decoding: Enhancing Multimodal Understanding in Audio-Visual Large Language Models

Chaeyoung Jung, Youngjoon Jang, Jongmin Choi, Joon Son Chung

机构 * Korea Advanced Institute of Science and Technology (KAIST)（韩国科学技术院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

HDA-SELD: Hierarchical Cross-Modal Distillation with Multi-Level Data Augmentation for Low-Resource Audio-Visual Sound Event Localization and Detection

Qing Wang, Ya Jiang, Hang Chen, Sabato Marco Siniscalchi, Jun Du, Jianqing Gao

机构 * University of Science and Technology of China（中国科学技术大学） ； University of Palermo（巴勒莫大学） ； iFLYTEK Research（iFLYTEK研究院） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM

Comments 13 pages, 8 figures

DEL: Dense Event Localization for Multi-modal Audio-Visual Understanding

Mona Ahmadian, Amir Shirian, Frank Guerin, Andrew Gilbert

机构 * University of Surrey（塞拉利昂大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title)；audio-visual(title)；multimodal(abstract)；cross-modal(abstract)

Multimodal Assessment of Speech Impairment in ALS Using Audio-Visual and Machine Learning Approaches

Francesco Pierotti, Andrea Bandini

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments Submitted to Interspeech

Locality-aware Cross-modal Correspondence Learning for Dense Audio-Visual Events Localization

Ling Xing, Hongyu Qu, Rui Yan, Xiangbo Shu, Jinhui Tang

机构 * School of Computer Science and Engineering, Nanjing University of Science and Technology（计算机科学与工程学院，南京理工大学） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

BeMERC: Behavior-Aware MLLM-based Framework for Multimodal Emotion Recognition in Conversation

Yumeng Fu, Junjie Wu, Zhongjie Wang, Meishan Zhang, Yulin Wu, Bingquan Liu

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(title,abstract)；分类 cs.CL

M$^3$AV: A Multimodal, Multigenre, and Multipurpose Audio-Visual Academic Lecture Dataset

Zhe Chen, Heyang Liu, Wenyi Yu, Guangzhi Sun, Hongcheng Liu, Ji Wu, Chao Zhang, Yu Wang, Yanfeng Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CL

Comments ACL 2024 Main Conference. Project website: https://jack-zc8.github.io/M3AV-dataset-page

Audio-Visual Generalized Zero-Shot Learning using Pre-Trained Large Multi-Modal Models

David Kurzendörfer, Otniel-Bogdan Mercea, A. Sophia Koepke, Zeynep Akata

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Comments CVPRw 2024 (L3D-IVU)

The Multimodal Information Based Speech Processing (MISP) 2023 Challenge: Audio-Visual Target Speaker Extraction

Shilong Wu, Chenxi Wang, Hang Chen, Yusheng Dai, Chenyue Zhang, Ruoyu Wang, Hongbo Lan, Jun Du, Chin-Hui Lee, Jingdong Chen, Shinji Watanabe, Sabato Marco Siniscalchi, Odette Scharenborg, Zhong-Qiu Wang, Jia Pan, Jianqing Gao

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments 5 pages, 4 figures

Multimodal Imbalance-Aware Gradient Modulation for Weakly-supervised Audio-Visual Video Parsing

Jie Fu, Junyu Gao, Changsheng Xu

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Audio-Visual Speech Enhancement and Separation by Utilizing Multi-Modal Self-Supervised Embeddings

I-Chun Chern, Kuo-Hsuan Hung, Yi-Ting Chen, Tassadaq Hussain, Mandar Gogate, Amir Hussain, Yu Tsao, Jen-Cheng Hou

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments ICASSP AMHAT 2023

The Multimodal Information based Speech Processing (MISP) 2022 Challenge: Audio-Visual Diarization and Recognition

Zhe Wang, Shilong Wu, Hang Chen, Mao-Kui He, Jun Du, Chin-Hui Lee, Jingdong Chen, Shinji Watanabe, Sabato Siniscalchi, Odette Scharenborg, Diyuan Liu, Baocai Yin, Jia Pan, Jianqing Gao, Cong Liu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；multi-modal(abstract)；分类 cs.MM

Comments 5 pages, 4 figures, to be published in ICASSP2023