多模态大模型

Information Fusion in Attention Networks Using Adaptive and Multi-level Factorized Bilinear Pooling for Audio-visual Emotion Recognition

Hengshun Zhou, Jun Du, Yuanyuan Zhang, Qing Wang, Qing-Feng Liu, Chin-Hui Lee

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM、eess.AS

The MuSe 2021 Multimodal Sentiment Analysis Challenge: Sentiment, Emotion, Physiological-Emotion, and Stress

Lukas Stappen, Alice Baird, Lukas Christ, Lea Schumann, Benjamin Sertolli, Eva-Maria Messner, Erik Cambria, Guoying Zhao, Björn W. Schuller

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CL、cs.AI

3D-MOV: Audio-Visual LSTM Autoencoder for 3D Reconstruction of Multiple Objects from Video

Justin Wilson, Ming C. Lin

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、eess.AS

Leveraging Recent Advances in Deep Learning for Audio-Visual Emotion Recognition

Liam Schoneveld, Alice Othmani, Hazem Abdelkawy

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments 8 pages, 3 figures, Pattern Recognition Letters

Multi-modal Residual Perceptron Network for Audio-Video Emotion Recognition

Xin Chang, Władysław Skarbek

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.AI、eess.AS

Multi-level Attention Fusion Network for Audio-visual Event Recognition

Mathilde Brousmiche, Jean Rouat, Stéphane Dupont

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Comments Preprint submitted to the Information Fusion journal in August 2020

Where and When: Space-Time Attention for Audio-Visual Explanations

Yanbei Chen, Thomas Hummel, A. Sophia Koepke, Zeynep Akata

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Tie Your Embeddings Down: Cross-Modal Latent Spaces for End-to-end Spoken Language Understanding

Bhuvan Agrawal, Markus Müller, Martin Radfar, Samridhi Choudhary, Athanasios Mouchtaris, Siegfried Kunzmann

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CL、eess.AS

Comments 7 pages, 6 figures

Seeing voices and hearing voices: learning discriminative embeddings using cross-modal self-supervision

Soo-Whan Chung, Hong Goo Kang, Joon Son Chung

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Under submission as a conference paper

Rule-embedded network for audio-visual voice activity detection in live musical video streams

Yuanbo Hou, Yi Deng, Bilei Zhu, Zejun Ma, Dick Botteldooren

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

Comments Submitted to ICASSP 2021

Themes Informed Audio-visual Correspondence Learning

Runze Su, Fei Tao, Xudong Liu, Haoran Wei, Xiaorong Mei, Zhiyao Duan, Lei Yuan, Ji Liu, Yuying Xie

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM

Comments Submitting to ICASSP 2021

SoundSpaces: Audio-Visual Navigation in 3D Environments

Changan Chen, Unnat Jain, Carl Schissler, Sebastia Vicenc Amengual Gari, Ziad Al-Halah, Vamsi Krishna Ithapu, Philip Robinson, Kristen Grauman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments Accepted to ECCV 2020 (Spotlight). Project page: http://vision.cs.utexas.edu/projects/audio_visual_navigation/

Audio-Visual Event Localization via Recursive Fusion by Joint Co-Attention

Bin Duan, Hao Tang, Wei Wang, Ziliang Zong, Guowei Yang, Yan Yan

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、eess.AS

Automatic Quality Assessment for Audio-Visual Verification Systems. The LOVe submission to NIST SRE Challenge 2019

Grigory Antipov, Nicolas Gengembre, Olivier Le Blouch, Gaël Le Lan

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM、eess.AS

Comments 5 pages, 1 figure, accepted at INTERSPEECH 2020. Corrected the reference [20]

Temporal aggregation of audio-visual modalities for emotion recognition

Andreea Birhala, Catalin Nicolae Ristea, Anamaria Radoi, Liviu Cristian Dutu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Disentangled Speech Embeddings using Cross-modal Self-supervision

Arsha Nagrani, Joon Son Chung, Samuel Albanie, Andrew Zisserman

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments ICASSP 2020. The first three authors contributed equally to this work

Look, Listen, and Act: Towards Audio-Visual Embodied Navigation

Chuang Gan, Yiwei Zhang, Jiajun Wu, Boqing Gong, Joshua B. Tenenbaum

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments Accepted by ICRA 2020. Project page: http://avn.csail.mit.edu

Audio-Visual Model Distillation Using Acoustic Images

Andrés F. Pérez, Valentina Sanguineti, Pietro Morerio, Vittorio Murino

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、eess.AS

Comments Accepted at WACV 2020; supplementary material at page 11; code available at https://github.com/afperezm/acoustic-images-distillation

Listening while Speaking and Visualizing: Improving ASR through Multimodal Chain

Johanes Effendi, Andros Tjandra, Sakriani Sakti, Satoshi Nakamura

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Accepted in IEEE ASRU 2019

Contextual Audio-Visual Switching For Speech Enhancement in Real-World Environments

Ahsan Adeel, Mandar Gogate, Amir Hussain

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、eess.AS

Comments 16 pages, 7 figures. arXiv admin note: substantial text overlap with arXiv:1808.00046

Journal ref Information Fusion, 2019

Weakly Supervised Representation Learning for Unsynchronized Audio-Visual Events

Sanjeel Parekh, Slim Essid, Alexey Ozerov, Ngoc Q. K. Duong, Patrick Pérez, Gaël Richard

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、eess.AS

Multimodal Utterance-level Affect Analysis using Visual, Audio and Text Features

Didan Deng, Yuqian Zhou, Jimin Pi, Bertram E. Shi

专题命中 音频语音多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments 5 pages, 1 figure, subject to the 2018 IJCNN challenge on One-Minute Gradual-Emotion Recognition

Do Visual Features Improve Other-Initiated Repair Detection? A Dyadic Multimodal Approach

视觉特征能否改善他人发起的修复检测？一种二元多模态方法

Anh Ngo, Nicolas Rollet, Catherine Pelachaud, Chloé Clavel

机构 * INRIA Paris（法国国家信息与自动化研究所巴黎分院） ； ISIR, Sorbonne University Paris（巴黎索邦大学信息学、系统与机器人研究所） ； Télécom Paris, SES, Institut Polytechnique de Paris, I3-CNRS Paris（巴黎电信学院、巴黎高等理工学院SES系、巴黎综合理工学院I3 - 法国国家科学研究中心） ； CNRS（法国国家科学研究中心） ； LTCI, Institut Polytechnique de Paris（巴黎综合理工学院LTCI实验室） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

AI总结 研究他人发起的修复检测问题，提出结合视觉特征的二元多模态模型，通过在两个语料库上评估，证明视觉信息能提升检测性能，提供跨模态特征贡献见解。

Comments Accepted to ICMI 2026 (International Conference on Multimodal Interaction), October 5-9, 2026, Napoli, Italy

Skating-Mixer: Long-Term Sport Audio-Visual Modeling with MLPs

Jingfei Xia, Mingchen Zhuge, Tiantian Geng, Shun Fan, Yuantai Wei, Zhenyu He, Feng Zheng

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Our code is available at https://github.com/AndyFrancesco29/Audio-Visual-Figure-Skating

Audio Visual Language Maps for Robot Navigation

Chenguang Huang, Oier Mees, Andy Zeng, Wolfram Burgard

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；multimodal foundation model(abstract)

Comments Project page: https://avlmaps.github.io/

SAMOT: State-Aware Step Modulation and Optimal Transport Matching for Audio-Visual Instance Segmentation

SAMOT：面向视听实例分割的状态感知步长调制与最优传输匹配

Kai Peng, Yunzhe Shen, Miao Zhang, Leiye Liu, Wei Ji, Jingjing Li, Yongri Piao, Huchuan Lu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.MM

AI总结 针对AVIS的模态状态变化干扰与跨模态分布差异问题，提出含ADSM与OT-MM的SAMOT框架，在AVIS基准取得最优性能。

Comments Accepted by ACM MM 2026

Whence the Voice? Self-supervised Dual-source Audio-Visual Localisation via Selective Convergence

声音从何而来？通过选择性收敛实现自监督双源视听定位

Han Hu, Dongheng Lin, Yuqi Hou, Haotian Li, Hyung Jin Chang, Jianbo Jiao

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM

AI总结 本文针对双源视听定位的循环依赖问题，利用自监督学习的选择性收敛提出两阶段框架，在双源基准上取得自监督方法最优性能，还通过引入像素级掩码修正了基准评估的不一致性。

C$^3$PO: Evaluating Cross-Modal Composition and Counterfactual Performance in Omnimodal Models

C³PO：评估全模态模型中的跨模态组合与反事实性能

Swapnanil Mukherjee, Agyeya Negi, Tanuja Ganu, Ponnurangam Kumaraguru

机构 * Microsoft Research India（微软研究院印度分院） ； IIIT Hyderabad（印度国际信息技术研究院（海得拉巴）） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.AI

AI总结 本研究推出C³PO基准数据集，评估全模态模型的跨模态组合与反事实性能，发现模型存在模态主导问题，需改进架构以支持持续跨模态注意力。

Beyond Boundary Frames: Talking-Head Inbetweening via Context-Aware Motion Modeling

超越边界框架：基于音频视觉语义的上下文中心视频插值

Yuchen Deng, Hai-Tao Zheng, Jie Wang, Xiaotian Li, Feidiao Yang, Yuxing Han

机构 * Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院） ； Pengcheng Laboratory（鹏城实验室） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

AI总结 本文提出BBF框架，通过解耦多模态条件，结合视觉上下文、文本语义和音频时间动态，提升视频插值的可控性和准确性，实验表明其在通用插值和音视频同步生成任务中优于现有方法。

Do We Really Need Multimodal Emotion Language Models Larger Than 1B Parameters?

我们真的需要参数超过10亿的多模态情感语言模型吗？

Kaiwen Zheng, Junchen Fu, Wenhao Deng, Hu Han, Joemon M. Jose, Xuri Ge

机构 * University of Glasgow（格拉斯哥大学） ； Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所） ； School of Artificial Intelligence, Shandong University（山东大学人工智能学院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

AI总结 质疑多模态情感识别需参数超10亿模型的假设，提出轻量级MER框架Light-MER，通过知识蒸馏及两种优化策略，在九个基准数据集实验中实现最优性能并显著提高推理效率，凸显小模型潜力。

Comments Accepted by ACM MM2026