多模态信息融合 - arXivDaily 专题

2406.13384 2024-06-21 cs.SD cs.CV cs.MM eess.AS 62%

Straight Through Gumbel Softmax Estimator based Bimodal Neural Architecture Search for Audio-Visual Deepfake Detection

Aravinda Reddy PN, Raghavendra Ramachandra, Krothapalli Sreenivasa Rao, Pabitra Mitra, Vinod Rathod

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

2403.18715 2024-06-06 cs.CV cs.AI cs.CL cs.MM 62%

Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding

Xintong Wang, Jingheng Pan, Liang Ding, Chris Biemann

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments Accepted to Findings of ACL 2024

2402.17187 2024-04-18 eess.IV cs.CV 62%

PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction

Zhaoxin Guo, Zhipeng Wang, Ruiquan Ge, Jianxun Yu, Feiwei Qin, Yuan Tian, Yuqing Peng, Yonghong Li, Changmiao Wang

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、eess.IV

2401.04154 2024-01-10 cs.CV cs.AI cs.LG cs.MM cs.SD eess.AS 62%

Efficient Selective Audio Masked Multimodal Bottleneck Transformer for Audio-Video Classification

Wentao Zhu

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments Accepted by WACV 2024; well-formatted PDF is in https://drive.google.com/file/d/1qvW52lamsvNGMCqPS7q8g8L4NaR_LlbR/view?usp=sharing. arXiv admin note: text overlap with arXiv:2401.04023

2210.04135 2023-10-31 cs.CV cs.LG cs.MM 62%

VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment

Shraman Pramanick, Li Jing, Sayan Nag, Jiachen Zhu, Hardik Shah, Yann LeCun, Rama Chellappa

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；分类 cs.CV、cs.MM

Comments Published in TMLR 2023

2305.11019 2023-10-10 cs.CV cs.AI cs.MM 62%

Annotation-free Audio-Visual Segmentation

Jinxiang Liu, Yu Wang, Chen Ju, Chaofan Ma, Ya Zhang, Weidi Xie

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

Comments Camera-ready version for WACV 2024; project page is https://jinxiang-liu.github.io/anno-free-AVS/

2309.16569 2023-09-29 cs.SD cs.CV cs.MM eess.AS 62%

Audio-Visual Speaker Verification via Joint Cross-Attention

R. Gnana Praveen, Jahangir Alam

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

2309.06176 2023-09-13 cs.CV cs.MM 62%

Dual-Path Temporal Map Optimization for Make-up Temporal Video Grounding

Jiaxiu Li, Kun Li, Jia Li, Guoliang Chen, Dan Guo, Meng Wang

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；分类 cs.CV、cs.MM

2309.04734 2023-09-12 cs.CV cs.CL cs.MM 62%

Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering

Yifan Dong, Suhang Wu, Fandong Meng, Jie Zhou, Xiaoli Wang, Jianxin Lin, Jinsong Su

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；分类 cs.CV、cs.MM

Comments Accepted In Proceedings of the 31st ACM International Conference on Multimedia (MM' 23)

2307.10813 2023-07-21 cs.CV cs.SD eess.AS eess.IV 62%

Perceptual Quality Assessment of Omnidirectional Audio-visual Signals

Xilei Zhu, Huiyu Duan, Yuqin Cao, Yuxin Zhu, Yucheng Zhu, Jing Liu, Li Chen, Xiongkuo Min, Guangtao Zhai

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、eess.IV

Comments 12 pages, 5 figures, to be published in CICAI2023

2306.10567 2023-06-21 eess.AS cs.CV cs.MM cs.SD 62%

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

Yuchen Hu, Chen Chen, Ruizhe Li, Heqing Zou, Eng Siong Chng

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments 14 pages, 5 figures, Accepted by ACL 2023

2305.01836 2023-05-04 cs.CV cs.LG cs.MM cs.SD eess.AS 62%

AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation

Shentong Mo, Yapeng Tian

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

2207.04213 2023-03-07 cs.MM cs.CV cs.LG cs.SD eess.AS 62%

Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extraction

Zhongweiyang Xu, Xulin Fan, Mark Hasegawa-Johnson

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

Comments Paper Accepted by ICASSP2023

2212.01040 2022-12-05 cs.CV cs.MM cs.SD eess.AS 62%

Role of Audio in Audio-Visual Video Summarization

Ibrahim Shoer, Berkay Kopru, Engin Erzin

专题命中音视频/视觉语言融合：information fusion(abstract)；分类 cs.CV、cs.MM

2104.08510 2021-04-26 cs.MM cs.CV cs.SD eess.AS 62%

Exploring Deep Learning for Joint Audio-Visual Lip Biometrics

Meng Liu, Longbiao Wang, Kong Aik Lee, Hanyi Zhang, Chang Zeng, Jianwu Dang

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

2009.04107 2020-09-10 eess.AS cs.MM cs.SD eess.IV 62%

Multi-modal Attention for Speech Emotion Recognition

Zexu Pan, Zhaojie Luo, Jichen Yang, Haizhou Li

专题命中音视频/视觉语言融合：hybrid fusion(abstract)；分类 eess.IV、cs.MM

Comments Accepted by Interspeech2020

2004.09476 2020-04-21 cs.CV cs.LG cs.MM cs.SD eess.AS 62%

Music Gesture for Visual Sound Separation

Chuang Gan, Deng Huang, Hang Zhao, Joshua B. Tenenbaum, Antonio Torralba

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

Comments CVPR 2020. Project page: http://music-gesture.csail.mit.edu

1809.10961 2019-10-30 cs.CV cs.MM stat.ML 62%

Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers

Yutong Ban, Xavier Alameda-Pineda, Laurent Girin, Radu Horaud

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV、cs.MM

1906.02497 2019-07-30 cs.IR cs.CV cs.MM 62%

Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos

Zhu Zhang, Zhijie Lin, Zhou Zhao, Zhenxin Xiao

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；分类 cs.CV、cs.MM

Comments Accepted by SIGIR 2019 as a full paper

Journal ref SIGIR, 2019, pages 655-664

1809.07999 2018-09-24 cs.CV cs.AI cs.CL cs.MM 62%

Multimodal Dual Attention Memory for Video Story Question Answering

Kyung-Min Kim, Seong-Ho Choi, Jin-Hwa Kim, Byoung-Tak Zhang

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV、cs.MM

Comments Accepted for ECCV 2018

2508.16300 2025-08-25 cs.CV cs.AI 61%

A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension

Mohammad Zia Ur Rehman, Devraj Raghuvanshi, Umang Jain, Shubhi Bansal, Nagendra Kumar

机构 * Indian Institute of Technology Indore（印度印度理工学院Indore）； Brown University（布朗大学）

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV；information fusion(comments)

Comments Published in Information Fusion

2607.15779 2026-08-12 cs.CV 版本更新 57%

Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion

通过交叉注意力和门控融合进行多模态矛盾与犹豫识别

Oussama Berhili, Yassine Ouzar, Larbi Boubchir

机构 * University of Paris 8（巴黎第八大学）； LIASD Laboratory（LIASD实验室）

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV

AI总结为ECCV 2026的ABAW11挑战赛开发多模态框架，用预训练编码器提取多模态特征，建立单模态基线，在此基础上提出含交叉注意力和门控融合的多模态架构，验证集宏F1达0.7394，提升显著。

2604.23957 2026-08-11 cs.CV 版本更新 57%

LAVA: Layered Audio-Visual Anti-tampering Watermarking for Robust Deepfake Detection and Localization

LAVA：分层音频视觉抗篡改水印用于鲁棒深度伪造检测与定位

Bokang Zeng, Zheng Gao, Xiaoyu Li, Xiaoyan Feng, Jiaojiao Jiang

机构 * School of Computer Science and Engineering（计算机科学与工程学院）； UNSW Sydney（新南威尔士大学悉尼分校）； School of Information and Communication Technology（信息与通信技术学院）； Griffith University（格里菲斯大学）

专题命中音视频/视觉语言融合：audio-visual fusion(abstract)；分类 cs.CV

AI总结 LAVA通过跨模态水印融合与校准感知对齐，提升深度伪造篡改检测与定位的鲁棒性，实验表明其检测性能接近完美，抗压缩与多模态错位能力强。

Comments Accepted at the 34th ACM International Conference on Multimedia (ACM MM 2026)

2509.16609 2026-08-11 cs.CV 版本更新 57%

Describe-to-Score: A text-guided framework for image complexity assessment

描述到评分：一种用于图像复杂度评估的文本引导框架

Shipeng Liu, Liang Zhao, Dengfeng Chen, Zhonglin Zhang

机构 * Xi’an University of Architecture and Technology（西安建筑科技大学）

专题命中音视频/视觉语言融合：multi-modal fusion(abstract)；分类 cs.CV

AI总结该研究针对现有图像复杂度评估方法无法捕捉高级语义的问题，提出文本引导框架D2S，仅训练阶段引入描述语义正则化，在IC9600基准达SOTA，跨任务也具竞争力。

Comments Accepted by Pattern Recognition

2608.06943 2026-08-10 cs.CV 新提交 57%

Dual-Space Modality Consistency Learning for Universal Cross-Modal Re-Identification

用于通用跨模态行人重识别的双空间模态一致性学习

Yujian Zhao, Yukang Zhao, Hankun Liu, Haoxuan Xu, Bo Li, Hanzi Wan, Guanglin Niu

机构 * School of Artificial Intelligence, Beihang University（北京航空航天大学人工智能学院）； School of Humanities and Social Sciences, Beihang University（北京航空航天大学人文与社会科学高等研究院）； School of Computer Science and Engineering, Beihang University（北京航空航天大学计算机科学与工程学院）； School of Informatics, Xiamen University（厦门大学信息学院）

专题命中音视频/视觉语言融合：visible-infrared(abstract)；分类 cs.CV

AI总结提出DSMCL即插即用框架，联合建模空间与频域一致性，在5个数据集的17项协议上提升跨模态ReID基线性能，适配多样化异构模态场景。

2608.03475 2026-08-05 cs.MM cs.AI cs.CL 新提交 57%

Adaptive Modality Reliability Diagnosis and Restoration for Robust Multimodal Intent Recognition

用于鲁棒多模态意图识别的自适应模态可靠性诊断与恢复

Suraj Kumar, Mohnish Raj, Soumi Chattopadhayay, Chandranath Adak, Ayan Dutta

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.MM

AI总结提出PRIME闭环可靠性引导框架，通过诊断恢复多模态意图识别的不可靠模态，在保持干净数据性能的同时提升了多模态数据各类退化场景下的鲁棒性。

2608.00043 2026-08-04 eess.SP cs.AI 新提交 57%

Multimodal Wearable-Based Olfactory-Induced Emotion Recognition in Arousal-Valence Dimensions

基于多模态可穿戴设备的嗅觉诱导唤醒-效价维度情感识别

Chen-Yang Xu, Lan Zhang, Fei-Yi Fan, Bin Hu, Qing-Hao Meng

专题命中音视频/视觉语言融合：hybrid fusion(abstract)；分类 eess.SP

AI总结针对现有嗅觉情感研究的维度失衡与多模态数据不足问题，构建多模态嗅觉情感数据集，提出STF-HFNet模型，在两个数据集上取得最优情感识别准确率。

2608.00986 2026-08-04 cs.CV 新提交 57%

Understanding and Overcoming Cross-modal Fusion Bias in Multimodal Anomaly Detection From A Fisher Information Perspective

从费舍尔信息视角理解和克服多模态异常检测中的跨模态融合偏差

Kaifang Long, Lianbo Ma, Liming Liu, Guoyang Xie

机构 * Northeastern University（东北大学）； CATL（宁德时代）

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV

AI总结该研究针对多模态异常检测中被忽视的跨模态融合偏差问题，提出即插即用框架UCFB，经实验验证在多种设置下均实现性能提升。

2607.28375 2026-07-31 cs.AI cs.MM 新提交 57%

HyperClaim: Fine-Grained Cross-Modal Hypergraph Reasoning for Video Misinformation Detection

HyperClaim：用于视频虚假信息检测的细粒度跨模态超图推理

Xiangbo Wang, Jiasheng Zhang, Xingtong Yu, Luoqiang Lei, Delvin Ce Zhang

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.MM

AI总结 HyperClaim是一种用于视频虚假信息检测的判别式时间超图框架，通过细粒度跨模态超图推理，在FactGuard协议下的三个数据集上准确率优于基线方法。

Comments 13 pages, including supplementary material

2607.26395 2026-07-30 cs.CV 新提交 57%

Registration-Grounded Spectral Fusion for Unregistered WLI/NBI Endoscopic Lesion Segmentation

基于配准的光谱融合用于未配准的WLI/NBI内镜病灶分割

Pengyu Jie, Wanquan Liu, Rui He, Pengcheng Li, Weiping Wen, Deyu Meng, Junwei Han, Chenqiang Gao

机构 * The First Affiliated Hospital of Sun Yat-sen University（中山大学附属第一医院）； School of Computer Science and Technology, Hainan University（海南大学计算机科学与技术学院）； School of Mathematics and Statistics, Xi’an Jiaotong University（西安交通大学数学与统计学院）； Chongqing University of Posts and Telecommunications（重庆邮电大学）

专题命中音视频/视觉语言融合：multimodal fusion(abstract)；分类 cs.CV

AI总结针对未配准的WLI/NBI内镜病灶分割问题，提出可靠性感知复域融合框架，通过拓扑正则化特征对应与可靠性引导的选择性融合，提升病灶分割性能。

Comments 11 pages

1. 音视频/视觉语言融合 495 篇

Straight Through Gumbel Softmax Estimator based Bimodal Neural Architecture Search for Audio-Visual Deepfake Detection

Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding

PE-MVCNet: Multi-view and Cross-modal Fusion Network for Pulmonary Embolism Prediction

Efficient Selective Audio Masked Multimodal Bottleneck Transformer for Audio-Video Classification

VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment

Annotation-free Audio-Visual Segmentation

Audio-Visual Speaker Verification via Joint Cross-Attention

Dual-Path Temporal Map Optimization for Make-up Temporal Video Grounding

Towards Better Multi-modal Keyphrase Generation via Visual Entity Enhancement and Multi-granularity Image Noise Filtering

Perceptual Quality Assessment of Omnidirectional Audio-visual Signals

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation

Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extraction

Role of Audio in Audio-Visual Video Summarization

Exploring Deep Learning for Joint Audio-Visual Lip Biometrics

Multi-modal Attention for Speech Emotion Recognition

Music Gesture for Visual Sound Separation

Variational Bayesian Inference for Audio-Visual Tracking of Multiple Speakers

Cross-Modal Interaction Networks for Query-Based Moment Retrieval in Videos

Multimodal Dual Attention Memory for Video Story Question Answering

A Multimodal-Multitask Framework with Cross-modal Relation and Hierarchical Interactive Attention for Semantic Comprehension

Multimodal Ambivalence and Hesitancy Recognition via Cross-Attention and Gated Fusion

LAVA: Layered Audio-Visual Anti-tampering Watermarking for Robust Deepfake Detection and Localization

Describe-to-Score: A text-guided framework for image complexity assessment

Dual-Space Modality Consistency Learning for Universal Cross-Modal Re-Identification

Adaptive Modality Reliability Diagnosis and Restoration for Robust Multimodal Intent Recognition

Multimodal Wearable-Based Olfactory-Induced Emotion Recognition in Arousal-Valence Dimensions

Understanding and Overcoming Cross-modal Fusion Bias in Multimodal Anomaly Detection From A Fisher Information Perspective

HyperClaim: Fine-Grained Cross-Modal Hypergraph Reasoning for Video Misinformation Detection

Registration-Grounded Spectral Fusion for Unregistered WLI/NBI Endoscopic Lesion Segmentation