多模态信息融合 - arXivDaily 专题

2209.09068 2022-09-20 cs.CV 74%

Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention

R Gnana Praveen, Eric Granger, Patrick Cardinal

专题命中音视频/视觉语言融合：audio-visual fusion(title)；分类 cs.CV

Comments arXiv admin note: substantial text overlap with arXiv:2203.14779, arXiv:2111.05222

2203.12367 2022-04-19 cs.CV 74%

Transformer-based Multimodal Information Fusion for Facial Expression Analysis

Wei Zhang, Feng Qiu, Suzhen Wang, Hao Zeng, Zhimeng Zhang, Rudong An, Bowen Ma, Yu Ding

专题命中音视频/视觉语言融合：information fusion(title)；分类 cs.CV

2104.06004 2021-11-05 cs.SD eess.AS eess.SP 74%

Detecting Escalation Level from Speech with Transfer Learning and Acoustic-Lexical Information Fusion

Ziang Zhou, Yanze Xu, Ming Li

专题命中音视频/视觉语言融合：information fusion(title)；分类 eess.SP

2105.14430 2021-06-01 cs.CV 74%

Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing

Jianning Wu, Zhuqing Jiang, Shiping Wen, Aidong Men, Haiying Wang

专题命中音视频/视觉语言融合：multimodal fusion(title)；分类 cs.CV

1803.04840 2018-03-14 cs.CV 74%

Resource aware design of a deep convolutional-recurrent neural network for speech recognition through audio-visual sensor fusion

Matthijs Van keirsbilck, Bert Moons, Marian Verhelst

专题命中音视频/视觉语言融合：sensor fusion(title)；分类 cs.CV

Comments Tech. report

1604.02946 2016-11-23 cs.CV 74%

Kernel-based Sensor Fusion with Application to Audio-Visual Voice Activity Detection

David Dov, Ronen Talmon, Israel Cohen

专题命中音视频/视觉语言融合：sensor fusion(title)；分类 cs.CV

1609.05281 2016-09-20 cs.CV 74%

GeThR-Net: A Generalized Temporally Hybrid Recurrent Neural Network for Multimodal Information Fusion

Ankit Gandhi, Arjun Sharma, Arijit Biswas, Om Deshmukh

专题命中音视频/视觉语言融合：information fusion(title)；分类 cs.CV

Comments To appear in ECCV workshop on Computer Vision for Audio-Visual Media, 2016

1105.2770 2015-03-18 cs.SD cs.MM 74%

Improving Performance of Speaker Identification System Using Complementary Information Fusion

Md. Sahidullah, Sandipan Chakroborty, Goutam Saha

专题命中音视频/视觉语言融合：information fusion(title)；分类 cs.MM

Comments 6 Pages, 3 figures

Journal ref Proceedings of 17th International Conference on Advanced Computing and Communications (ADCOM 2009) pp. 182-187 (2009)

2605.08723 2026-05-12 cs.CV cs.MM 73%

EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing

EAR：增强单模表示以实现弱监督音频视觉视频解析

Huilai Li, Xiaomeng Di, Ying Xing, Yonghao Dang, Yiming Wang, Jianqin Yin

机构 * School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications（智能工程与自动化学院，北京邮电大学）； State Grid Corporation of China（国家电网公司）； School of Artificial Intelligence, Beijing University of Posts and Telecommunications（人工智能学院，北京邮电大学）

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；audio-visual fusion(abstract)；分类 cs.CV、cs.MM

AI总结本文提出EAR框架，通过增强伪标签生成器和AVVP模型的单模表示，提升视频解析的时序定位性能。

2602.00701 2026-02-03 cs.MM cs.CV cs.LG cs.SD 73%

Cross-Modal Binary Attention: An Energy-Efficient Fusion Framework for Audio-Visual Learning

跨模态二进制注意力：面向音频视觉学习的高效融合框架

Mohamed Saleh, Zahra Ahmadi

机构 * Peter L. Reichertz Institute for Medical Informatics of TU Braunschweig and Hannover Medical School（图林根工业大学和汉诺威医学院医学信息学研究所）； Lower Saxony Center for AI and Causal Methods in Medicine (CAIMed)（下萨克森人工智能与因果医学方法中心）

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；audio-visual fusion(abstract)；分类 cs.CV、cs.MM

AI总结提出CMQKA和SNNergy，通过高效二进制操作实现线性复杂度的跨模态融合，显著提升音频视觉任务的能耗效率和性能

2308.08143 2024-02-05 cs.SD cs.CV cs.MM eess.AS 73%

IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation

Kai Li, Runxuan Yang, Fuchun Sun, Xiaolin Hu

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；multi-modal fusion(abstract)；分类 cs.CV、cs.MM

Comments 18 pages, 6 figures

2603.25538 2026-08-04 cs.LG cs.SE 版本更新 71%

ARMOR: A Robust Self-Supervised Framework for Root Cause Analysis in Microservices under Missing Modality

面向缺失数据的多模态融合用于统一微服务故障管理

Wenzhuo Qian, Hailiang Zhao, Ziqi Wang, Zhipeng Gao, Jiayi Chen, Zhiwei Ling, Shuiguang Deng

机构 * Zhejiang University（浙江大学）

专题命中音视频/视觉语言融合：multimodal fusion(title)

AI总结本文提出ARMOR框架，通过自监督学习解决微服务故障管理中多模态数据缺失问题，提升异常检测、故障分类和根本原因定位的性能。

Comments Accepted by the Proceedings of the 41st IEEE/ACM International Conference on Automated Software Engineering (ASE '26), 2026. This is the authors' version of the work; the definitive Version of Record is forthcoming

2603.22372 2026-07-16 cs.LG cs.AI 版本更新 71%

Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion

重新思考时间序列的多模态融合：文本模态需要受约束的融合

Seunghan Lee, Jun Seo, Jaehoon Lee, Sungdong Yoo, Minjae Kim, Tae Yoon Lim, Dongwan Kang, Hwanil Choi, SoonYoung Lee, Wonbin Ahn

机构 * LG AI Research（LG人工智能研究）

专题命中音视频/视觉语言融合：multimodal fusion(title)

AI总结针对多模态时间序列预测中朴素融合方法效果不佳的问题，提出受约束融合方法及受控融合适配器（CFA），通过低秩适配器过滤无关文本信息，在多种数据集和模型上验证了有效性。

Comments KDD Workshop on Mining and Learning from Time Series 2026 (Oral presentation)

2601.02438 2026-01-21 cs.SE cs.AI cs.CR 71%

Focus on What Matters: Fisher-Guided Adaptive Multimodal Fusion for Vulnerability Detection

聚焦关键要素：基于Fisher信息的自适应多模态融合用于漏洞检测

Yun Bian, Yi Chen, HaiQuan Wang, ShiHao Li, Zhe Cui

机构 * University of Chinese Academy of Sciences（中国科学院大学）

专题命中音视频/视觉语言融合：multimodal fusion(title)

AI总结本文提出TaCCS-DFA框架，通过Fisher信息引导的自适应多模态融合，提升漏洞检测的F1分数，同时降低推理延迟。

2506.08936 2025-06-11 cs.LG 71%

BioLangFusion: Multimodal Fusion of DNA, mRNA, and Protein Language Models

Amina Mollaysa, Artem Moskale, Pushpak Pati, Tommaso Mansi, Mangal Prakash, Rui Liao

专题命中音视频/视觉语言融合：multimodal fusion(title)

Comments Proceedings of ICML 2025 Workshop on Multi-modal Foundation Proceedings of ICML 2025 Workshop on Multi-modal Foundation Proceedings of ICML 2025 Workshop on Multi-modal Foundation Models and Large Language Models for Life Sciences

2506.02088 2025-06-04 cs.SD cs.CL cs.LG 71%

Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025

Alef Iury Siqueira Ferreira, Lucas Rafael Gris, Alexandre Ferro Filho, Lucas Ólives, Daniel Ribeiro, Luiz Fernando, Fernanda Lustosa, Rodrigo Tanaka, Frederico Santos de Oliveira, Arlindo Galvão Filho

机构 * Federal University of Goiás（戈亚斯联邦大学）； Federal University of Rio Grande do Norte（北里奥格兰德联邦大学）； Aeronautics Institute of Technology（航空技术研究所）； Federal University of Mato Grosso（马托格罗索联邦大学）

专题命中音视频/视觉语言融合：multimodal fusion(title)

2501.16621 2025-01-29 cs.LG cs.AI 71%

Chinese Stock Prediction Based on a Multi-Modal Transformer Framework: Macro-Micro Information Fusion

Lumen AI, Tengzhou No. 1 Middle School, Shihao Ji, Zihui Song, Fucheng Zhong, Jisen Jia, Zhaobo Wu, Zheyi Cao, Xu Tianhao

专题命中音视频/视觉语言融合：information fusion(title)

2211.01299 2023-09-28 eess.AS cs.CL cs.SD 71%

Late Audio-Visual Fusion for In-The-Wild Speaker Diarization

Zexu Pan, Gordon Wichern, François G. Germain, Aswin Subramanian, Jonathan Le Roux

专题命中音视频/视觉语言融合：audio-visual fusion(title)

2305.16406 2023-08-24 cs.CL 71%

Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech

Loukas Ilias, Dimitris Askounis

专题命中音视频/视觉语言融合：multimodal fusion(title)

Comments Knowledge-Based Systems

Journal ref Knowledge-Based Systems (Volume: 277, October 2023)

2302.05894 2023-04-06 cs.LG 71%

Neural Architecture Search with Multimodal Fusion Methods for Diagnosing Dementia

Michail Chatzianastasis, Loukas Ilias, Dimitris Askounis, Michalis Vazirgiannis

专题命中音视频/视觉语言融合：multimodal fusion(title)

Comments Accepted at ICASSP 2023

2211.00526 2022-11-02 cs.CL cs.AI 71%

Leveraging Graph-based Cross-modal Information Fusion for Neural Sign Language Translation

Jiangbin Zheng, Siyuan Li, Cheng Tan, Chong Wu, Yidong Chen, Stan Z. Li

专题命中音视频/视觉语言融合：information fusion(title)

2202.04528 2022-09-19 cs.SD cs.LG eess.AS 71%

Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement

Leandro Aparecido Passos, João Paulo Papa, Javier Del Ser, Amir Hussain, Ahsan Adeel

专题命中音视频/视觉语言融合：information fusion(title)

2110.00385 2021-10-05 cs.NE cs.AI cs.LG 71%

Neural Dependency Coding inspired Multimodal Fusion

Shiv Shankar

专题命中音视频/视觉语言融合：multimodal fusion(title)

2106.09668 2021-06-18 cs.LG 71%

Multi-modal fusion with gating using audio, lexical and disfluency features for Alzheimer's Dementia recognition from spontaneous speech

Morteza Rohanian, Julian Hough, Matthew Purver

专题命中音视频/视觉语言融合：multi-modal fusion(title)

Journal ref Proc. Interspeech 2020, 2187-2191

2607.25543 2026-07-29 cs.CV cs.AI 新提交 70%

Less is More: Modality-Decoupling for General AIGC Audio-Video Detection

少即是多：通用AIGC音频-视频检测的模态解耦

Jielun Peng, Yabin Wang, Yaqi Li, Jincheng Liu, Xiaopeng Hong, Athanasios V. Vasilakos

机构 * Harbin Institute of Technology（哈尔滨工业大学）； University of Agder（阿格德大学）

专题命中音视频/视觉语言融合：feature-level fusion(abstract)；decision-level fusion(abstract)；分类 cs.CV

AI总结针对通用AIGC音频-视频检测，现有方法假设不总成立，本文提出DAV-Det系统，通过决策级融合独立建模各模态取证证据，视觉和音频检测器分别利用多粒度表示及门控双分支架构，在挑战赛中排名第一。

Comments First place in the General AIGC Audio-Video Detection Challenge at the IJCAI-ECAI 2026 DDL 2.0 Workshop

2607.16327 2026-07-21 cs.CV 新提交 70%

Localization-Infused Vision-Language Semantic Fusion for Text-Guided Medical Image Segmentation

用于文本引导医学图像分割的定位注入视觉语言语义融合

Songyue Han, Mingye Zou, Shuchang Ye, Lei Bi, Mingyuan Meng

机构 * Air Force Engineering University（空军工程大学）； Harbin Institute of Technology（哈尔滨工业大学）； University of Sydney（悉尼大学）； Institute of Translational Medicine, Shanghai Jiao Tong University（上海交通大学转化医学研究院）； Zhongguancun Institute of Artificial Intelligence（中关村人工智能研究院）

专题命中音视频/视觉语言融合：information fusion(abstract)；feature-level fusion(abstract)；分类 cs.CV

AI总结研究针对文本引导医学图像分割，提出LoG框架，通过联合执行多尺度目标定位任务，实现三级定位注入语义融合，在三个基准数据集实验中表现出色，优于现有医学图像分割方法。

2603.13056 2026-03-16 cs.CV cs.AI 70%

Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach

团队RAS在第10届ABAW竞赛中的表现：多模态valence和arousal估计方法

Elena Ryumina, Maxim Markitantov, Alexandr Axyonov, Dmitry Ryumin, Mikhail Dolgushin, Denis Dresvyanskiy, Alexey Karpov

机构 * St. Petersburg Federal Research Center of the Russian Academy of Sciences（俄罗斯科学院圣彼得堡联邦研究中心）； ITMO University（ITMO大学）

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；audio-visual fusion(abstract)；分类 cs.CV

AI总结本文提出一种多模态方法，用于在真实环境中估计valence和arousal。结合面部、行为和音频模态，利用GRADA、Transformer、Qwen3-VL-4B-Instruct和Mamba等技术，实现跨模态融合，最终在Aff-Wild2数据集上达到0.658的CCC值。

Comments 8 pages, 1 figure

2503.09873 2025-03-14 cs.CV 70%

FDCT: Frequency-Aware Decomposition and Cross-Modal Token-Alignment for Multi-Sensor Target Classification

Shoaib Meraj Sami, Md Mahedi Hasan, Nasser M. Nasrabadi, Raghuveer Rao

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；image fusion(abstract)；分类 cs.CV

Comments 12 pages Accepted in the IEEE Transactions on Aerospace and Electronic Systems

2409.07078 2024-09-12 cs.CV cs.AI 70%

Multimodal Emotion Recognition with Vision-language Prompting and Modality Dropout

Anbin QI, Zhongliang Liu, Xinyong Zhou, Jinba Xiao, Fengrun Zhang, Qi Gan, Ming Tao, Gaozheng Zhang, Lu Zhang

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；information fusion(abstract)；分类 cs.CV

2403.04654 2024-04-29 cs.CV cs.SD eess.AS 70%

Audio-Visual Person Verification based on Recursive Fusion of Joint Cross-Attention

R. Gnana Praveen, Jahangir Alam

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；feature-level fusion(abstract)；分类 cs.CV

Comments Accepted to FG2024

1. 音视频/视觉语言融合 495 篇

Audio-Visual Fusion for Emotion Recognition in the Valence-Arousal Space Using Joint Cross-Attention

Transformer-based Multimodal Information Fusion for Facial Expression Analysis

Detecting Escalation Level from Speech with Transfer Learning and Acoustic-Lexical Information Fusion

Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing

Resource aware design of a deep convolutional-recurrent neural network for speech recognition through audio-visual sensor fusion

Kernel-based Sensor Fusion with Application to Audio-Visual Voice Activity Detection

GeThR-Net: A Generalized Temporally Hybrid Recurrent Neural Network for Multimodal Information Fusion

Improving Performance of Speaker Identification System Using Complementary Information Fusion

EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing

Cross-Modal Binary Attention: An Energy-Efficient Fusion Framework for Audio-Visual Learning

IIANet: An Intra- and Inter-Modality Attention Network for Audio-Visual Speech Separation

ARMOR: A Robust Self-Supervised Framework for Root Cause Analysis in Microservices under Missing Modality

Rethinking Multimodal Fusion for Time Series: Text Modalities Need Constrained Fusion

Focus on What Matters: Fisher-Guided Adaptive Multimodal Fusion for Vulnerability Detection

BioLangFusion: Multimodal Fusion of DNA, mRNA, and Protein Language Models

Enhancing Speech Emotion Recognition with Graph-Based Multimodal Fusion and Prosodic Features for the Speech Emotion Recognition in Naturalistic Conditions Challenge at Interspeech 2025

Chinese Stock Prediction Based on a Multi-Modal Transformer Framework: Macro-Micro Information Fusion

Late Audio-Visual Fusion for In-The-Wild Speaker Diarization

Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech

Neural Architecture Search with Multimodal Fusion Methods for Diagnosing Dementia

Leveraging Graph-based Cross-modal Information Fusion for Neural Sign Language Translation

Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement

Neural Dependency Coding inspired Multimodal Fusion

Multi-modal fusion with gating using audio, lexical and disfluency features for Alzheimer's Dementia recognition from spontaneous speech

Less is More: Modality-Decoupling for General AIGC Audio-Video Detection

Localization-Infused Vision-Language Semantic Fusion for Text-Guided Medical Image Segmentation

Team RAS in 10th ABAW Competition: Multimodal Valence and Arousal Estimation Approach

FDCT: Frequency-Aware Decomposition and Cross-Modal Token-Alignment for Multi-Sensor Target Classification

Multimodal Emotion Recognition with Vision-language Prompting and Modality Dropout

Audio-Visual Person Verification based on Recursive Fusion of Joint Cross-Attention