多模态大模型

DoubleHelix: Structured Cross-Modal Fusion for Audio-Visual Speech Recognition with LLMs

DoubleHelix：结合大语言模型的视听语音识别结构化跨模态融合方法

Ziwei Cheng, Zhenhua Tan, Zhuomin Zhu

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI

AI总结 该研究针对视听语音识别跨模态融合缺乏结构化迭代优化的问题，提出DoubleHelix融合框架，通过三个组件实现迭代交互与退化感知增强，在LRS3数据集上大幅降低词错误率并提升噪声鲁棒性。

Comments ACM MM2026 ACCEPTED

EVAS: Efficient Multimodal Temporal Forgery Localization via Audio-Visual Synergy and Steered Boundary Calibration

EVAS：通过视听协同和引导边界校准实现高效多模态时间伪造定位

Shen Shen, Quan Zhang, Dan Jiang, Ke Zhang

机构 * Soochow University（苏州大学） ； Tsinghua University（清华大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

AI总结 针对人工智能生成内容带来的多模态取证挑战，提出EVAS框架，利用多阶段视听协同机制和边界感知细化策略，结合解耦训练范式与轻量级网络，有效定位长视频中稀疏分布的伪造片段。

Agentic Active Omni-Modal Perception for Multi-Hop Audio-Visual Reasoning

面向多跳音视频推理的主动全模态感知代理

Ke Xu, Yuhao Wang, Ziyang Cheng, Hongcheng Liu, Yanfeng Wang, Yu Wang

机构 * Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；omni-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI

AI总结 针对多跳音视频推理中证据稀疏且跨模态分布的问题，提出MOV-Bench基准和AOP-Agent代理框架，通过分层全模态记忆与观察-反思-重规划循环实现主动感知，显著提升开源全模态大模型在长视频和推理密集型问题上的性能。

Attribution-Guided Multimodal Deepfake Detection via Cross-Modal Forensic Fingerprints

基于属性引导的多模态深度伪造检测：跨模态取证指纹

Wasim Ahmad, Wei Zhang, Xuerui Mao

机构 * School of Interdisciplinary Science, Beijing Institute of Technology, Beijing 100081, China（交叉学科学院，北京理工大学，北京100081，中国） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

AI总结 本文提出AMDD框架，通过跨模态取证指纹一致性损失，引导模型学习生成器特定的取证特征，提升多模态深度伪造检测的准确性和鲁棒性。

AVID: A Benchmark for Omni-Modal Audio-Visual Inconsistency Understanding via Agent-Driven Construction

AVID：一种通过代理驱动构建的多模态音频视觉不一致理解基准

Zixuan Chen, Depeng Wang, Hao Lin, Li Luo, Ke Xu, Ya Guo, Huijia Zhu, Tanfeng Sun, Xinghao Jiang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；omni-modal(title,abstract)；cross-modal(abstract)；分类 cs.MM

AI总结 AVID是首个大规模多模态音频视觉不一致理解视频基准，通过可扩展的构建流程和8种细粒度不一致类别，评估检测、时间定位、分类和推理能力，验证了其在可信多模态AI系统中的有效性。

A Systematic Study of Cross-Modal Typographic Attacks on Audio-Visual Reasoning

多模态字体攻击在音频视觉推理中的系统研究

Tianle Chen, Deepti Ghadiyaram

机构 * Boston University（波士顿大学） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

AI总结 本文研究多模态字体攻击对多模态大语言模型的影响，发现跨模态攻击比单模态攻击更有效，揭示了多模态推理中的关键安全漏洞。

Multimodal Self-Attention Network with Temporal Alignment for Audio-Visual Emotion Recognition

多模态自注意力网络与时间对齐用于音频-视觉情感识别

Inyong Koo, yeeun Seong, Minseok Son, Jaehyuk Jang, Changick Kim

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.MM

AI总结 本文提出基于Transformer的多模态自注意力网络，通过时间对齐旋转位置嵌入和跨时间匹配损失，解决音频-视觉情感识别中的跨模态帧率不匹配问题，提升时间线索保留和跨模态融合效果。

Comments 5 pages, 3 figures, accepted to ICASSP 2026

MAD: Modality-Adaptive Decoding for Mitigating Cross-Modal Hallucinations in Multimodal Large Language Models

MAD：用于减轻多模态大语言模型中跨模态幻觉的模态自适应解码

Sangyun Chung, Se Yeon Kim, Youngchae Chee, Yong Man Ro

机构 * Integrated Vision Language Lab, KAIST, South Korea（韩国科学技术院集成视觉语言实验室） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.AI

AI总结 MAD通过自适应加权对比解码分支，有效减少多模态大语言模型中的跨模态幻觉问题。

Audio-Visual Cross-Modal Compression for Generative Face Video Coding

音频-视觉跨模态压缩用于生成式面部视频编码

Youmin Xu, Mengxi Guo, Shijie Zhao, Weiqi Li, Junlin Li, Li Zhang, Jian Zhang

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM

AI总结 本文提出AVCC框架，通过联合压缩音频和视频流，提升生成式面部视频编码的率-失真性能。

Comments Accepted as a PAPER and for publication in the DCC 2026 proceedings

Are Multimodal Foundation Models All That Is Needed for Emofake Detection?

Mohd Mujtaba Akhtar, Girish, Orchid Chetia Phukan, Swarup Ranjan Behera, Pailla Balakrishna Reddy, Ananda Chandra Nayak, Sanjib Kumar Nayak, Arun Balaji Buduru

专题命中 音频语音多模态 ：multimodal(title,abstract)；multimodal foundation model(title,abstract)；cross-modal(abstract)；分类 eess.AS

Comments Accepted to APSIPA-ASC 2025

Improving Noise Robust Audio-Visual Speech Recognition via Router-Gated Cross-Modal Feature Fusion

DongHoon Lim, YoungChae Kim, Dong-Hyun Kim, Da-Hee Yang, Joon-Hyuk Chang

机构 * Dept. Artificial Intelligence（人工智能系） ； Hanyang University（翰阳大学） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted to IEEE ASRU 2025

AVHBench: A Cross-Modal Hallucination Benchmark for Audio-Visual Large Language Models

Kim Sung-Bin, Oh Hyun-Bin, JungMok Lee, Arda Senocak, Joon Son Chung, Tae-Hyun Oh

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments ICLR 2025

Enhancing Audio-Visual Spiking Neural Networks through Semantic-Alignment and Cross-Modal Residual Learning

Xiang He, Dongcheng Zhao, Yiting Dong, Guobin Shen, Xin Yang, Yi Zeng

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments The manuscript is under review and the code is available https://github.com/Brain-Cog-Lab/S-CMRL

Omni-Emotion: Extending Video MLLM with Detailed Face and Audio Modeling for Multimodal Emotion Analysis

Qize Yang, Detao Bai, Yi-Xing Peng, Xihan Wei

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(title,abstract)；multi-modal(abstract)；分类 cs.CV

Audio-Visual Approach For Multimodal Concurrent Speaker Detection

Amit Eliav, Sharon Gannot

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；cross-modal(abstract)；分类 eess.AS

Comments 12 pages

AVS-Mamba: Exploring Temporal and Multi-modal Mamba for Audio-Visual Segmentation

Sitong Gong, Yunzhi Zhuge, Lu Zhang, Yifan Wang, Pingping Zhang, Lijun Wang, Huchuan Lu

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted to IEEE Transactions on Multimedia (TMM)

Cross-modal Cognitive Consensus guided Audio-Visual Segmentation

Zhaofeng Shi, Qingbo Wu, Fanman Meng, Linfeng Xu, Hongliang Li

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；multi-modal(abstract)；分类 eess.AS

Comments Accepted by IEEE Transactions on Multimedia; 16 pages

Contextual Cross-Modal Attention for Audio-Visual Deepfake Detection and Localization

Vinaya Sree Katamneni, Ajita Rattani

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(title)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

CAT: Enhancing Multimodal Large Language Model to Answer Questions in Dynamic Audio-Visual Scenarios

Qilang Ye, Zitong Yu, Rui Shao, Xinyu Xie, Philip Torr, Xiaochun Cao

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；MLLM(abstract)；分类 cs.CV

Rethink Cross-Modal Fusion in Weakly-Supervised Audio-Visual Video Parsing

Yating Xu, Conghui Hu, Gim Hee Lee

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments WACV 2024

Temporal and cross-modal attention for audio-visual zero-shot learning

Otniel-Bogdan Mercea, Thomas Hummel, A. Sophia Koepke, Zeynep Akata

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments ECCV 2022

Leveraging Unimodal Self-Supervised Learning for Multimodal Audio-Visual Speech Recognition

Xichen Pan, Peiyu Chen, Yichen Gong, Helong Zhou, Xinbing Wang, Zhouhan Lin

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL2022 Main Conference

Dynamic Content Moderation in Livestreams: Combining Supervised Classification with MLLM-Boosted Similarity Matching

直播中的动态内容审核：结合监督分类与MLLM增强的相似度匹配

Wei Chee Yew, Hailun Xu, Sanjay Saha, Xiaotian Fan, Hiok Hian Ong, David Yuchen Wang, Kanchan Sarkar, Zhenheng Yang, Danhui Guan

机构 * TikTok Singapore Singapore（TikTok新加坡） ； TikTok San Jose United States（TikTok旧金山美国） ； TikTok Shanghai China（TikTok上海中国） 纠错

专题命中 音频语音多模态 ：MLLM(title,title_cn)；multimodal(abstract)；分类 cs.CV、cs.AI

AI总结 提出一种混合审核框架，结合监督分类和基于参考的相似度匹配，利用多模态大语言模型提升准确性，在保持轻量推理的同时实现大规模直播内容审核。

Comments To be published at KDD 2026 (ADS track)

Rationale-Guided Learning for Multimodal Emotion Recognition

基于理由引导学习的多模态情感识别

Sujung Oh, Jung Uk Kim, Sangmin Lee

专题命中 音频语音多模态 ：MLLM(summary_cn,abstract)；multimodal(title,abstract)；分类 cs.AI

AI总结 针对多模态情感识别忽略人类因果推理的问题，提出理由引导学习框架，结合双加工理论与MLLM生成的理由训练模型，在IEMOCAP和MELD基准取得最优性能，且推理无额外开销。

Comments ICASSP 2026

Journal ref S. Oh, J. U. Kim and S. Lee, "Rationale-Guided Learning for Multimodal Emotion Recognition," ICASSP 2026 - 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), Barcelona, Spain, 2026, pp. 12577-12581

Music Audio-Visual Question Answering Requires Specialized Multimodal Designs

音乐音频视觉问答需要专门的多模态设计

Wenhao You, Xingjian Diao, Wenjun Huang, Chunhui Zhang, Keyi Kong, Weiyi Wu, Chiyu Ma, Zhongyu Ouyang, Tingxuan Wu, Ming Cheng, Soroush Vosoughi, Jiang Gui

机构 * University of Waterloo（滑铁卢大学） ； Dartmouth College（达特茅斯学院） ； UC Irvine（加州大学尔湾分校） ； New York University（纽约大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

AI总结 本文探讨了音乐音频视觉问答任务中多模态设计的必要性，指出需专门的输入处理、空间时间架构和音乐特定建模策略以应对连续密集的音频视觉内容和复杂时间动态。

Comments Accepted to Annual Meeting of the Association for Computational Linguistics (ACL 2026). The first two authors contributed equally

Integrating Fine-Grained Audio-Visual Evidence for Robust Multimodal Emotion Reasoning

融合细粒度音频视觉证据以实现鲁棒的多模态情绪推理

Zhixian Zhao, Wenjie Tian, Lei Xie

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

AI总结 SABER-LLM通过构建大规模情绪推理数据集和结构化证据分解范式，实现鲁棒的多模态情绪推理。

Adaptive Audio-Visual Speech Recognition via Matryoshka-Based Multimodal LLMs

Umberto Cappellazzo, Minsu Kim, Stavros Petridis

机构 * Imperial College London（伦敦帝国学院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to IEEE ASRU 2025

MMS-LLaMA: Efficient LLM-based Audio-Visual Speech Recognition with Minimal Multimodal Speech Tokens

Jeong Hun Yeo, Hyeongseop Rha, Se Jin Park, Yong Man Ro

机构 * Integrated Vision and Language Lab, KAIST, South Korea（集成视觉与语言实验室，韩国科学技术院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at Findings of ACL 2025. The code and models are available https://github.com/JeongHun0716/MMS-LLaMA

Zero-Shot Audio-Visual Editing via Cross-Modal Delta Denoising

Yan-Bo Lin, Kevin Lin, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Chung-Ching Lin, Xiaofei Wang, Gedas Bertasius, Lijuan Wang

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Project page: https://genjib.github.io/project_page/AVED/index.html

Pilot-guided Multimodal Semantic Communication for Audio-Visual Event Localization

Fei Yu, Zhe Xiang, Nan Che, Zhuoran Zhang, Yuandi Li, Junxiao Xue, Zhiguo Wan

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS