多模态信息融合

Attentive Fusion Enhanced Audio-Visual Encoding for Transformer Based Robust Speech Recognition

Liangfa Wei, Jie Zhang, Junfeng Hou, Lirong Dai

专题命中 音视频/视觉语言融合 ：information fusion(abstract)；audio-visual fusion(abstract)；分类 cs.MM

AMBER: An Adaptive Multimodal Mask Transformer for Beam Prediction with Missing Modalities

AMBER: 一种自适应多模态掩码变换器用于缺失模态的波束预测

Chenyiming Wen, Binpu Shi, Min Li, Ming-Min Zhao, Min-Jian Zhao, Jiangzhou Wang

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；feature-level fusion(abstract)

AI总结 AMBER通过自适应处理多模态数据中的缺失模态问题，提升波束预测的鲁棒性和准确性，实验表明其在真实世界数据集上优于现有多模态学习基线。

Comments 13 pages, 9 figures

Audio-visual cross-modality knowledge transfer for machine learning-based in-situ monitoring in laser additive manufacturing

Jiarui Xie, Mutahar Safdar, Lequn Chen, Seung Ki Moon, Yaoyao Fiona Zhao

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；audio-visual fusion(abstract)

Comments 47 pages, 19 figures, 6 tables

A Survey of IMU Based Cross-Modal Transfer Learning in Human Activity Recognition

Abhi Kamboj, Minh Do

专题命中 音视频/视觉语言融合 ：sensor fusion(abstract)；分类 cs.CV、eess.IV、eess.SP

Multi-Modal Emotion Recognition by Text, Speech and Video Using Pretrained Transformers

Minoo Shayaninasab, Bagher Babaali

专题命中 音视频/视觉语言融合 ：feature-level fusion(abstract)；decision-level fusion(abstract)

CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation with Transformers

Jiaming Zhang, Huayao Liu, Kailun Yang, Xinxin Hu, Ruiping Liu, Rainer Stiefelhagen

专题命中 音视频/视觉语言融合 ：multi-modal fusion(abstract)；分类 cs.CV、eess.IV、cs.RO

Comments Accepted to IEEE Transactions on Intelligent Transportation Systems (T-ITS). The source code of CMX is publicly available at https://github.com/huaaaliu/RGBX_Semantic_Segmentation

Multimodal Prompt Transformer with Hybrid Contrastive Learning for Emotion Recognition in Conversation

Shihao Zou, Xianying Huang, Xudong Shen

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；information fusion(abstract)

Comments Accepted to ACM MM 2023

When did you become so smart, oh wise one?! Sarcasm Explanation in Multi-modal Multi-party Dialogues

Shivani Kumar, Atharva Kulkarni, Md Shad Akhtar, Tanmoy Chakraborty

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；information fusion(abstract)

Comments Accepted in ACL 2022. 13 pages, 4 figures, 12 tables

Event-VLA: Action-Conditioned Event Fusion for Robust Vision-Language-Action Model

Event-VLA: 基于动作条件的事件融合用于鲁棒的视觉-语言-动作模型

Jiaxin Liu, Xun Xu, Zhenhao Zhang, Hanqing Wang, Ruiqi Chen, Shi Chang, Weiyu Guo, Laurent Kneip

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、cs.RO

AI总结 针对现有VLA模型在光照变化下鲁棒性不足的问题，提出Event-VLA框架，通过事件流作为光照鲁棒的运动敏感互补观测，利用动作查询路由和门控交叉注意力融合事件信息，提升低光及近黑暗环境下的操作成功率。

Omni-Customizer: End-to-End MultiModal Customization for Joint Audio-Video Generation

Omni-Customizer: 用于联合音频-视频生成的端到端多模态定制

Yuheng Chen, Qingdong He, Teng Hu, Yuji Wang, Yabiao Wang, Lizhuang Ma, Jiangning Zhang

机构 * Shanghai Jiao Tong University（上海交通大学） ； Zhejiang University（浙江大学） ； University of Electronic Science and Technology of China（电子科技大学） 纠错

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、cs.MM

AI总结 本文提出Omni-Customizer，一种端到端多模态定制框架，旨在实现精确的多模态身份信息绑定和无缝融合，通过引入Omni-Context Fusion模块和Masked TTS Cross-Attention机制，提升多模态定制生成的性能。

SemiFA: An Agentic Multi-Modal Framework for Autonomous Semiconductor Failure Analysis Report Generation

SemiFA：一种用于自主半导体故障分析报告生成的代理多模态框架

Shivam Chand Kaushik

机构 * School of Artificial Intelligence and Data Engineering (SAIDE)（人工智能与数据工程学院） ； Indian Institute of Technology Jodhpur（印度理工学院贾尔普尔） 纠错

专题命中 音视频/视觉语言融合 ：multi-modal fusion(abstract)；分类 cs.CV、eess.IV

AI总结 SemiFA通过多代理语言图框架实现自主生成半导体故障分析报告，利用多模态数据融合提升根因分析精度，首次整合SECS/GEM设备 telemetry 进行自动化报告生成。

Comments 11 pages, 6 figures, 8 tables. Dataset available at https://huggingface.co/datasets/ShivamChand/SemiFA-930. Code available at https://github.com/Shivamckaushik/SemiFA

Residual Cross-Modal Fusion Networks for Audio-Visual Navigation

残差跨模态融合网络用于音频视觉导航

Yi Wang, Yinfeng Yu, Bin Ren

机构 * School of Computer Science and Technology, Xinjiang University, Urumqi, China（新疆大学计算机科学与技术学院） ； Joint International Research Laboratory of Silk Road Multilingual Cognitive（丝绸之路多语认知联合国际实验室） ； School of Mechatronic Engineering and Automation Shanghai University, Shanghai, China（上海大学机电工程与自动化学院） 纠错

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、cs.RO

AI总结 本文提出残差跨模态融合网络，通过双向残差交互实现音频视觉信息互补建模，提升跨域导航性能。

Comments Main paper (10 pages). Accepted for publication by the 14th international conference on Computational Visual Media (CVM 2026)

AdSum: Two-stream Audio-visual Summarization for Automated Video Advertisement Clipping

AdSum：用于自动化视频广告剪辑的双流音频视觉摘要

Wen Xie, Yanjun Zhu, Gijs Overgoor, Yakov Bart, Agata Lapedriza Garcia, Sarah Ostadabbas

机构 * Northeastern University（东北大学） ； Southern Methodist University（南方 Methodist 大学） 纠错

专题命中 音视频/视觉语言融合 ：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

AI总结 AdSum提出了一种双流音频视觉融合模型，用于自动化视频广告剪辑，通过重要性预测提升广告剪辑效率。

Comments Accepted at 32nd International Conference on MultiMedia Modeling

MCIHN: A Hybrid Network Model Based on Multi-path Cross-modal Interaction for Multimodal Emotion Recognition

Haoyang Zhang, Zhou Yang, Ke Sun, Yucai Pang, Guoliang Xu

机构 * Chongqing University of Posts and Telecommunications（重庆邮电大学） ； Xi’an Jiaotong University（西安交通大学） ； University of New South Wales（新南威尔士大学） 纠错

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments The paper will be published in the MMAsia2025 conference proceedings

Ovi: Twin Backbone Cross-Modal Fusion for Audio-Video Generation

Chetwin Low, Weimin Wang, Calder Katyal

机构 * Character AI ； Yale University（耶鲁大学） 纠错

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Integrating Spatial and Semantic Embeddings for Stereo Sound Event Localization in Videos

Davide Berghi, Philip J. B. Jackson

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 eess.IV、eess.SP

Comments arXiv admin note: substantial text overlap with arXiv:2507.04845

Seeing is Believing: Emotion-Aware Audio-Visual Language Modeling for Expressive Speech Generation

Weiting Tan, Jiachen Lian, Hirofumi Inaguma, Paden Tomasello, Philipp Koehn, Xutai Ma

机构 * Johns Hopkins University（约翰霍普金斯大学） ； Meta AI Research（Meta AI 研究） 纠错

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments EMNLP 2025 (Findings)

Think Before You Segment: An Object-aware Reasoning Agent for Referring Audio-Visual Segmentation

Jinxing Zhou, Yanghao Zhou, Mingfei Han, Tong Wang, Xiaojun Chang, Hisham Cholakkal, Rao Muhammad Anwer

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments Project page: https://github.com/jasongief/TGS-Agent

Localizing Audio-Visual Deepfakes via Hierarchical Boundary Modeling

Xuanjun Chen, Shih-Peng Cheng, Jiawei Du, Lin Zhang, Xiaoxiao Miao, Chung-Che Wang, Haibin Wu, Hung-yi Lee, Jyh-Shing Roger Jang

专题命中 音视频/视觉语言融合 ：audio-visual fusion(abstract)；分类 cs.CV、eess.IV

Comments Work in progress

Spatial and Semantic Embedding Integration for Stereo Sound Event Localization and Detection in Regular Videos

Davide Berghi, Philip J. B. Jackson

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 eess.IV、eess.SP

Research on Audio-Visual Quality Assessment Dataset and Method for User-Generated Omnidirectional Video

Fei Zhao, Da Pan, Zelu Qi, Ping Shi

机构 * School of Information and Communication Engineering（信息与通信工程学院） 纠错

专题命中 音视频/视觉语言融合 ：audio-visual fusion(abstract)；分类 cs.CV、eess.IV

Comments Our paper has been accepted by ICME 2025

Adaptive Prototype Learning for Multimodal Cancer Survival Analysis

Hong Liu, Haosen Yang, Federica Eduati, Josien P. W. Pluim, Mitko Veta

专题命中 音视频/视觉语言融合 ：information fusion(abstract)；分类 cs.CV、eess.IV

Comments 10 pages, 3 figures

Multimodal Action Quality Assessment

Ling-An Zeng, Wei-Shi Zheng

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、eess.SP

Comments IEEE Transactions on Image Processing 2024

Language-Guided Diffusion Model for Visual Grounding

Sijia Chen, Baochun Li

专题命中 音视频/视觉语言融合 ：multi-modal fusion(abstract)；分类 cs.CV、cs.MM

Comments 20 pages, 16 figures

MambaPlace:Text-to-Point-Cloud Cross-Modal Place Recognition with Attention Mamba Mechanisms

Tianyi Shang, Zhenyu Li, Pengjie Xu, Jinwei Qiao

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、cs.RO

Comments 8 pages

Multimodal Music Generation with Explicit Bridges and Retrieval Augmentation

Baisen Wang, Le Zhuo, Zhaokai Wang, Chenxi Bao, Wu Chengjing, Xuecheng Nie, Jiao Dai, Jizhong Han, Yue Liao, Si Liu

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、cs.MM

WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition

Feng Li, Jiusong Luo, Wanjun Xia

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments Accepted by 31st International Conference on MultiMedia Modeling (MMM2025)

Talking Head Generation Driven by Speech-Related Facial Action Units and Audio- Based on Multimodal Representation Fusion

Sen Chen, Zhilei Liu, Jiaxing Liu, Longbiao Wang

专题命中 音视频/视觉语言融合 ：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments arXiv admin note: text overlap with arXiv:2110.09951

EEPNet: Efficient Edge Pixel-based Matching Network for Cross-Modal Dynamic Registration between LiDAR and Camera

Yuanchao Yue, Hui Yuan, Suai Li, Qi Jiang

专题命中 音视频/视觉语言融合 ：sensor fusion(abstract)；分类 cs.CV、eess.IV

Enhancing Modal Fusion by Alignment and Label Matching for Multimodal Emotion Recognition

Qifei Li, Yingming Gao, Yuhua Wen, Cong Wang, Ya Li

专题命中 音视频/视觉语言融合 ：information fusion(abstract)；分类 cs.CV、cs.MM

Comments The paper has been accepted by INTERSPEECH 2024