多模态大模型 - arXivDaily 专题

2312.05412 2024-10-10 cs.LG cs.CV cs.MM cs.SD eess.AS 87%

CMMD: Contrastive Multi-Modal Diffusion for Video-Audio Conditional Modeling

Ruihan Yang, Hannes Gamper, Sebastian Braun

专题命中音频语音多模态：multi-modal(title,abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

2406.09833 2024-07-17 cs.AI cs.MM cs.SD eess.AS 87%

SHMamba: Structured Hyperbolic State Space Model for Audio-Visual Question Answering

Zhe Yang, Wenrui Li, Guanghui Cheng

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

2406.04930 2024-06-10 cs.CV cs.MM cs.SD eess.AS 87%

MA-AVT: Modality Alignment for Parameter-Efficient Audio-Visual Transformers

Tanvir Mahmud, Shentong Mo, Yapeng Tian, Diana Marculescu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted in Efficient Deep Learning for Computer Vision CVPR Workshop 2024

2308.08143 2024-02-05 cs.SD cs.CV cs.MM eess.AS 87%

IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation

Kai Li, Runxuan Yang, Fuchun Sun, Xiaolin Hu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 18 pages, 6 figures

2306.02858 2023-10-26 cs.CL cs.CV cs.SD eess.AS 87%

Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding

Hang Zhang, Xin Li, Lidong Bing

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Accepted by EMNLP 2023's demo track; Code, Pretrained Model, and Dataset: https://github.com/DAMO-NLP-SG/Video-LLaMA

2302.03533 2023-02-20 cs.CV cs.MM cs.SD eess.AS 87%

Revisiting Pre-training in Audio-Visual Learning

Ruoxuan Feng, Wenke Xia, Di Hu

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2212.03814 2022-12-09 cs.CV cs.MM cs.SD eess.AS 87%

iQuery: Instruments as Queries for Audio-Visual Sound Separation

Jiaben Chen, Renrui Zhang, Dongze Lian, Jiaqi Yang, Ziyao Zeng, Jianbo Shi

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2203.02216 2022-07-08 cs.SD cs.AI cs.MM eess.AS 87%

Look\&Listen: Multi-Modal Correlation Learning for Active Speaker Detection and Speech Enhancement

Junwen Xiong, Yu Zhou, Peng Zhang, Lei Xie, Wei Huang, Yufei Zha

专题命中音频语音多模态：multi-modal(title,abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 13 pages, 8figures

2007.10558 2020-07-23 cs.CV cs.MM cs.SD eess.AS 87%

Unified Multisensory Perception: Weakly-Supervised Audio-Visual Video Parsing

Yapeng Tian, Dingzeyu Li, Chenliang Xu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments ECCV 2020 (Spotlight)

1811.03875 2019-04-16 cs.CL cs.CV cs.LG eess.AS 87%

Multimodal One-Shot Learning of Speech and Images

Ryan Eloff, Herman A. Engelbrecht, Herman Kamper

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments 5 pages, 1 figure, 3 tables; accepted to ICASSP 2019

2608.09435 2026-08-11 cs.AI 新提交 86%

Listen, See and Track: Spatio-Temporal Audio-Visual Sound Event Reasoning for Omni-Modal Language Models

听、看与跟踪：面向全模态大模型的时空视听声音事件推理

Zhi Zeng, Cheng Zhang, Zesheng Yang, Rendong Pi, Jiaying Wu, Di Zhang, Zihan Ma, Guodong Li, Zhou Yang, Yu Xiang, Yifei Zheng, Minnan Luo

专题命中音频语音多模态：audio-visual(title,abstract)；omni-modal(title)；分类 cs.AI

AI总结针对现有模型缺失的时空视听推理能力，构建ST-OmniQA基准，提出ST-Omni-R1模型，在该基准上平均语义准确率达77.83%，且空间运动表示可跨基准迁移。

2603.01530 2026-06-30 cs.MM 86%

CueNet: Robust Audio-Visual Speaker Extraction through Cross-Modal Cue Mining and Interaction

CueNet：通过跨模态线索挖掘与交互实现鲁棒的音频视觉说话人提取

Jiadong Wang, Ke Zhang, Xinyuan Qian, Ruijie Tao, Haizhou Li, Björn Schuller

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(title)；分类 cs.MM

AI总结本文提出CueNet，通过跨模态线索挖掘与交互提升音频视觉说话人提取的鲁棒性，验证了模型在不同视觉退化下的优越性能。

2605.21008 2026-05-21 eess.AS 86%

A Survey of Audio Reasoning in Multimodal Foundation Models

多模态基础模型中音频推理的综述

Zhihan Guo, Wenqian Cui, Guan-Ting Lin, Daxin Tan, Jingyao Li, Qiyong Zheng, Dingdong Wang, Jing Xiong, Han Shi, Jiaya Jia, Irwin King

专题命中音频语音多模态：multimodal(title)；multimodal foundation model(title)；audio-visual(abstract)；分类 eess.AS

AI总结本文综述了多模态基础模型中音频推理的研究问题，探讨了音频推理模型的架构和训练基础，并系统整理了音频到文本、音频到语音、音频视觉推理和代理音频推理等领域的最新进展，同时分析了新兴范式和评估实践，提出了未来发展方向。

2505.18110 2026-02-03 cs.CL 86%

Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM

Watch and Listen: 通过多模态大语言模型理解音频-视觉-语音时刻

Zinuo Li, Xian Zhang, Yongxin Guo, Mohammed Bennamoun, Farid Boussaid, Girish Dwivedi, Luqi Gong, Qiuhong Ke

机构 * University of Western Australia（西澳大学）； Alibaba Group（阿里巴巴集团）； Zhejiang Laboratory（浙江实验室）； Monash University（墨尔本大学）

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title)；分类 cs.CL

AI总结 TriSense通过整合视觉、音频和语音模态，提升视频时间理解能力，采用基于查询的连接器实现多模态鲁棒性，并通过TriSense-2M数据集推动多模态视频分析发展。

Comments Accepted by NeurIPS 2025

2512.02759 2025-12-03 eess.AS cs.SD eess.IV 86%

Towards Language-Independent Face-Voice Association with Multimodal Foundation Models

面向语言无关的面容-语音关联的多模态基础模型

Aref Farhadipour, Teodora Vukovic, Volker Dellwo

机构 * Department of Computational Linguistics, University of Zurich（苏黎世大学计算语言学系）

专题命中音频语音多模态：multimodal(title)；multimodal foundation model(title)；cross-modal(abstract)；分类 eess.AS

AI总结本文提出了一种基于多模态基础模型的跨语言面容-语音关联方法，通过ImageBind与LoRA结合，实现了在未见过语言上的有效验证。

Comments This paper presents the system description of the UZH-CL team for the FAME2026 Challenge at ICASSP 2026. Our model achieved second place in the final ranking

2509.14592 2025-09-19 cs.MM cs.SD 86%

MMED: A Multimodal Micro-Expression Dataset based on Audio-Visual Fusion

Junbo Wang, Yan Zhao, Shuo Li, Shibo Wang, Shigang Wang, Jian Wei

机构 * College of Communication Engineering, Jilin University（吉林大学通信工程学院）

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title)；分类 cs.MM

2501.08085 2025-01-15 cs.CL cs.LG 86%

Dynamic Multimodal Sentiment Analysis: Leveraging Cross-Modal Attention for Enabled Classification

Hui Lee, Singh Suniljit, Yong Siang Ong

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(title)；分类 cs.CL

2401.17796 2024-02-01 cs.SD eess.AS 86%

Exploiting Audio-Visual Features with Pretrained AV-HuBERT for Multi-Modal Dysarthric Speech Reconstruction

Xueyuan Chen, Yuejiao Wang, Xixin Wu, Disong Wang, Zhiyong Wu, Xunying Liu, Helen Meng

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(title)；分类 eess.AS

Comments Accepted by ICASSP 2024

2310.03827 2023-10-09 cs.CV 86%

Integrating Audio-Visual Features for Multimodal Deepfake Detection

Sneha Muppalla, Shan Jia, Siwei Lyu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(title)；分类 cs.CV

2309.04628 2023-09-12 eess.AS cs.SD 86%

Leveraging Pretrained Image-text Models for Improving Audio-Visual Learning

Saurabhchand Bhati, Jesús Villalba, Laureano Moro-Velazquez, Thomas Thebaud, Najim Dehak

专题命中音频语音多模态：audio-visual(title,abstract)；image-text(title)；分类 eess.AS

2204.06228 2023-05-25 cs.CV 86%

Do You Really Mean That? Content Driven Audio-Visual Deepfake Dataset and Multimodal Method for Temporal Forgery Localization

Zhixi Cai, Kalin Stefanov, Abhinav Dhall, Munawar Hayat

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(title)；分类 cs.CV

Comments DICTA 2022

1701.04224 2017-03-20 cs.CV 86%

Auxiliary Multimodal LSTM for Audio-visual Speech Recognition and Lipreading

Chunlin Tian, Weijun Ji

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(title)；分类 cs.CV

Comments 8 pages, 4 figures

2604.15086 2026-08-14 cs.MM cs.CV cs.SD 版本更新 86%

ControlFoley: Unified and Controllable Video-to-Audio Generation with Cross-Modal Conflict Handling

ControlFoley: 一种统一且可控的视频到音频生成方法，具有跨模态冲突处理

Jianxuan Yang, Xinyue Guo, Zhi Cheng, Kai Wang, Lipan Zhang, Jinjie Hu, Qiang Ji, Yihua Cao, Yihao Meng, Zhaoyue Cui, Mengmei Liu, Meng Meng, Jian Luan

机构 * MiLM Plus, Xiaomi Inc.（小米MiLM Plus）； MiLM Plus, Xiaomi Inc. Wuhan University（小米MiLM Plus 武汉大学）； Wuhan University（武汉大学）

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM

AI总结 ControlFoley通过联合视觉编码和时域-音域解耦提升视频到音频生成的可控性与同步性，在多种任务中表现优异。

2605.27944 2026-08-12 cs.AI cs.MM cs.SD 版本更新 86%

From Talking to Singing: A New Challenge for Audio-Visual Deepfake Detection

从说话到唱歌：音视频深度伪造检测的新挑战

Ke Liu, Jiwei Wei, Wenyu Zhang, Shuchang Zhou, Ruikun Chai, Yutao Dai, Chaoning Zhang, Yang Yang

机构 * Center for Future Media, School of Computer Science and Engineering, University of Electronic Science and Technology of China（未来媒体中心，计算机科学与工程学院，电子科技大学）

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.AI、cs.MM

AI总结针对现有音视频深度伪造检测方法在唱歌场景中性能下降的问题，提出文本引导的音视频伪造检测框架（T-AVFD），通过面部真实性模式学习和多模态差异权重学习，在说话和唱歌场景中均实现鲁棒检测。

Comments Accepted by ICML 2026

2608.08794 2026-08-11 cs.AI cs.MM cs.SD 新提交 86%

Deferred Audio Pruning with Local Audio-Visual Dynamics for Omni-LLMs

面向全模态大语言模型（Omni-LLMs）的基于局部视听动态的延迟音频剪枝方法

Kyeongyoon Lee, Hongyeob Kim, Youngeun Kim, Sungeun Hong

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；omni-modal(abstract)；分类 cs.AI、cs.MM

AI总结针对全模态大语言模型的高开销问题，提出两阶段框架A-PACK，利用局部视听动态延迟音频剪枝，在Qwen2.5-Omni基准上实现性能最优，同时大幅降低预填充开销并提升解码吞吐量。

2607.16107 2026-07-20 eess.AS cs.CV 新提交 86%

Audio-Visual Flamingo: Open Audio-Visual Intelligence for Long and Complex Videos

视听火烈鸟：用于长而复杂视频的开放视听智能

Sreyan Ghosh, Arushi Goel, Kaousheik Jayakumar, Lasha Koroshinadze, Nishit Anand, Siddharth Gururani, Hanrong Ye, Pritam Biswas, Yuanhang Su, Ehsan Hosseini-Asl, Sang-gil Lee, Zhifeng Kong, Jaehyeon Kim, Sungwon Kim, S Sakshi, Ramani Duraiswami, Dinesh Manocha, Andrew Tao, Mohammad Shoeybi, Bryan Catanzaro, Ming-Yu Liu, Wei Ping

机构 * NVIDIA, USA（美国NVIDIA公司）； University of Maryland, USA（美国马里兰大学）

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；omni-modal(abstract)；分类 cs.CV、eess.AS

AI总结研究提出视听火烈鸟，用于长复杂视频的联合理解与推理。贡献包括构建大规模视频集、设计三阶段课程及推理框架。实验显示其在多基准测试中表现优异，超越同类开放模型，在长复杂视听理解推理任务中竞争力强，有现实应用价值和泛化能力。

Comments Project Page: https://avflamingo.pages.dev/

2607.15295 2026-07-20 cs.MM cs.AI cs.LG cs.SD 新提交 86%

AV-JEPA: Extending LeJEPA to Audio-Visual Self-Supervised Learning

AV-JEPA：将LeJEPA扩展到视听自监督学习

Benjamin Robson, Santeri Mentu, Wenshuai Zhao, Arno Solin

机构 * ELLIS Institute Finland（芬兰ELLIS研究所）； Department of Computer Science, Aalto University, Espoo, Finland（艾尔沃斯大学计算机科学系）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.AI、cs.MM

AI总结研究将LeJEPA扩展到视听自监督学习，核心方法是用早期融合视觉Transformer和模态丢弃作掩码训练模型对齐视图嵌入，主要贡献是实现跨模态对齐，架构简洁，在分类任务中有竞争力且支持零样本音频-视频检索。

2605.08723 2026-05-12 cs.CV cs.MM 86%

EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing

EAR：增强单模表示以实现弱监督音频视觉视频解析

Huilai Li, Xiaomeng Di, Ying Xing, Yonghao Dang, Yiming Wang, Jianqin Yin

机构 * School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications（智能工程与自动化学院，北京邮电大学）； State Grid Corporation of China（国家电网公司）； School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

AI总结本文提出EAR框架，通过增强伪标签生成器和AVVP模型的单模表示，提升视频解析的时序定位性能。

2604.16617 2026-04-21 cs.CV cs.MM cs.SD 86%

AVRT: Audio-Visual Reasoning Transfer through Single-Modality Teachers

AVRT：通过单模态教师模型实现音频-视觉推理迁移

Edson Araujo, Saurabhchand Bhati, M. Jehanzeb Mirza, Brian Kingsbury, Samuel Thomas, Rogerio Feris, James R. Glass, Hilde Kuehne

机构 * University of Tübingen, Germany（图宾根大学）； MIT, Cambridge MA, USA（麻省理工学院）； IBM Research, USA（IBM研究院）； MIT-IBM Watson AI Lab, USA（麻省理工-IBM沃森人工智能实验室）； Tuebingen AI Center, Germany（图宾根人工智能中心）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

AI总结本文提出AVRT框架，通过单模态教师模型生成高质量音频-视觉推理轨迹，并利用LLM合并模型整合轨迹，从而在多模态设置中实现推理迁移，取得音视频和音频任务的最优效果。

2604.11102 2026-04-14 cs.CV cs.MM 86%

OmniScript: Towards Audio-Visual Script Generation for Long-Form Cinematic Video

OmniScript: 向长篇影视视频的音频-视觉脚本生成迈进

Junfu Pu, Yuxin Chen, Teng Wang, Ying Shan

机构 * ARC Lab, Tencent（腾讯ARC实验室）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；omni-modal(abstract)；分类 cs.CV、cs.MM

AI总结本文提出视频到脚本任务，介绍OmniScript模型，通过渐进式训练在长篇叙事理解中实现高效脚本生成，优于开源模型并接近专有模型。

Comments Project Page: https://arcomniscript.github.io

1. 音频语音多模态 4549 篇