多模态大模型

Self-Supervised Audio-Visual Speech Representations Learning By Multimodal Self-Distillation

Jing-Xuan Zhang, Genshun Wan, Zhen-Hua Ling, Jia Pan, Jianqing Gao, Cong Liu

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments submitted to ICASSP 2023

Self-Supervised Audio-Visual Representation Learning with Relaxed Cross-Modal Synchronicity

Pritam Sarkar, Ali Etemad

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Comments Accepted in AAAI 2023

A Novel Frame Structure for Cloud-Based Audio-Visual Speech Enhancement in Multimodal Hearing-aids

Abhijeet Bishnu, Ankit Gupta, Mandar Gogate, Kia Dashtipour, Ahsan Adeel, Amir Hussain, Mathini Sellathurai, Tharmalingam Ratnarajah

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Multimodal Audio-Visual Information Fusion using Canonical-Correlated Graph Neural Network for Energy-Efficient Speech Enhancement

Leandro Aparecido Passos, João Paulo Papa, Javier Del Ser, Amir Hussain, Ahsan Adeel

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Multi-Modal Perception Attention Network with Self-Supervised Learning for Audio-Visual Speaker Tracking

Yidi Li, Hong Liu, Hao Tang

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by AAAI2022

Learning Explicit and Implicit Latent Common Spaces for Audio-Visual Cross-Modal Retrieval

Donghuo Zeng, Jianming Wu, Gen Hattori, Yi Yu, Rong Xu

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM

A Survey on Audio Synthesis and Audio-Visual Multimodal Processing

Zhaofeng Shi

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Attention-based cross-modal fusion for audio-visual voice activity detection in musical video streams

Yuanbo Hou, Zhesong Yu, Xia Liang, Xingjian Du, Bilei Zhu, Zejun Ma, Dick Botteldooren

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments Accepted by INTERSPEECH 2021

Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing

Jianning Wu, Zhuqing Jiang, Shiping Wen, Aidong Men, Haiying Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

VisualVoice: Audio-Visual Speech Separation with Cross-Modal Consistency

Ruohan Gao, Kristen Grauman

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Comments In CVPR 2021. Project page: http://vision.cs.utexas.edu/projects/VisualVoice/

Audio-Visual Instance Discrimination with Cross-Modal Agreement

Pedro Morgado, Nuno Vasconcelos, Ishan Misra

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Audio-Visual Speech Separation Using Cross-Modal Correspondence Loss

Naoki Makishima, Mana Ihori, Akihiko Takashima, Tomohiro Tanaka, Shota Orihashi, Ryo Masumura

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Comments Accepted to ICASSP 2021

Audio-visual Speaker Recognition with a Cross-modal Discriminative Network

Ruijie Tao, Rohan Kumar Das, Haizhou Li

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 eess.AS

Multimodal Integration for Large-Vocabulary Audio-Visual Speech Recognition

Wentao Yu, Steffen Zeiler, Dorothea Kolossa

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；multi-modal(abstract)；分类 eess.AS

Comments 5 pages

Journal ref Published in Proceedings of the 28th European Signal Processing Conference (EUSIPCO), 2020

Multi-modal Egocentric Activity Recognition using Audio-Visual Features

Mehmet Ali Arabacı, Fatih Özkan, Elif Surer, Peter Jančovič, Alptekin Temizel

专题命中 音频语音多模态 ：multi-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.CV

Journal ref Multimedia Tools and Applications (2020)

Audio-Visual Embedding for Cross-Modal MusicVideo Retrieval through Supervised Deep CCA

Donghuo Zeng, Yi Yu, Keizo Oyama

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM

Comments 8 pages, 9 figures. Accepted by ISM 2018

Audio-Visual Speech Enhancement Using Multimodal Deep Convolutional Neural Networks

Jen-Cheng Hou, Syu-Siang Wang, Ying-Hui Lai, Yu Tsao, Hsiu-Wen Chang, Hsin-Min Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.MM

Comments To appear in IEEE Transactions on Emerging Topics in Computational Intelligence. Some audio samples can be reached in this link: https://sites.google.com/view/avse2017

Deep Multimodal Learning for Audio-Visual Speech Recognition

Youssef Mroueh, Etienne Marcheret, Vaibhava Goel

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)；分类 cs.CL

Comments ICASSP 2015

LongAV-Compass: Towards Unified Evaluation of Minute-Scale Audio-Visual Generation Across T2AV, I2AV, and V2AV

LongAV-Compass：面向分钟级音视频生成在T2AV、I2AV和V2AV上的统一评估

Tengfei Liu, Yang Shi, Xuanyu Zhu, Jiafu Tang, Liu Yang, Qixun Wang, Zhuoran Zhang, Yuqi Tang, Fengxiang Wang, Yuhao Dong, Xinlong Chen, Bozhou Li, Bohan Zeng, Yue Ding, Xiaohan Zhang, Jialu Chen, Haotian Wang, Yuanxing Zhang, Pengfei Wan, Leye Wang

机构 * Peking University（北京大学） ； Kling Team（Kling团队） ； Nanjing University（南京大学） ； SJTU（上海交通大学） ； HKUST(GZ)（香港科技大学（广州）） ； Shanghai AI Lab（上海人工智能实验室） ； Nanyang Technological University（南洋理工大学） ； CASIA（中国科学院自动化所） ； Tsinghua University（清华大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；MLLM(abstract,abstract_cn)；multimodal(abstract)；分类 cs.CV、cs.MM

AI总结 针对现有评估协议局限于短片段的问题，提出LongAV-Compass基准，通过284个测试用例和统一评估框架，系统评估分钟级音视频生成在文本、图像、视频条件下的质量、一致性和对齐。

Beyond the Cartesian Illusion: Testing Two-Stage Multi-Modal Theory of Mind under Perceptual Bottlenecks

超越笛卡尔错觉：在感知瓶颈下测试双阶段多模态理论 of Mind

Yajing Zhou, Xiangyu Kong

机构 * College of Computer Science, Beijing Information Science and Technology University（北京信息科技大学计算机学院） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；MLLM(abstract,abstract_cn)；audio-visual(abstract)；分类 cs.CV、cs.AI

AI总结 本文研究了多模态大语言模型在感知瓶颈下的双阶段空间推理能力，提出了一种基于锚点的具身体验空间分解链式推理方法，以解决空间对称性和视角模糊性问题，从而提升多模态理论 of Mind 的表现。

Comments 17 pages, 3 figures

PMMTalk: Speech-Driven 3D Facial Animation from Complementary Pseudo Multi-modal Features

Tianshun Han, Shengnan Gui, Yiqing Huang, Baihui Li, Lijian Liu, Benjia Zhou, Ning Jiang, Quan Lu, Ruicong Zhi, Yanyan Liang, Du Zhang, Jun Wan

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；image-text(abstract)；audio-visual(abstract)

Asymmetric Hierarchical Anchoring for Robust Audio-Visual Cross-Modal Generalization

非对称层次锚定用于音频-视觉联合表示：解决信息分配模糊性以实现鲁棒跨模态泛化

Bixing Wu, Yuhong Zhao, Zongli Ye, Jiachen Lian, Xiangyu Yue, Gopala Anumanchipalli

机构 * Zhejiang University, China（浙江大学） ； University of California, Berkeley, USA（加州大学伯克利分校） ； MMLab, Chinese University of Hong Kong, China（香港中文大学MMLab） 纠错

专题命中 音频语音多模态 ：cross-modal(title,abstract)；audio-visual(title,abstract)

AI总结 本文提出非对称层次锚定框架，通过结构化语义锚点解决跨模态泛化中的信息分配模糊性，提升语义一致性和表示纯度。

AGAV-Rater: Adapting Large Multimodal Model for AI-Generated Audio-Visual Quality Assessment

Yuqin Cao, Xiongkuo Min, Yixuan Gao, Wei Sun, Guangtao Zhai

机构 * Institute of Image Communication ； Network Engineering, Shanghai Key Laboratory of Digital Media Processing ； Transmissions, Shanghai Jiao Tong University, Shanghai ； School of Communication \& Electronic Engineering, East China Normal University, Shanghai 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(title)；分类 cs.CV、cs.MM、eess.AS

Multimodal Trustworthy Semantic Communication for Audio-Visual Event Localization

Yuandi Li, Zhe Xiang, Fei Yu, Zhangshuang Guan, Hui Ji, Zhiguo Wan, Cheng Feng

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)

Dual-Path Cross-Modal Attention for better Audio-Visual Speech Extraction

Zhongweiyang Xu, Xulin Fan, Mark Hasegawa-Johnson

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(title)；分类 cs.CV、cs.MM、eess.AS

Comments Paper Accepted by ICASSP2023

Towards Multimodal Understanding of Passenger-Vehicle Interactions in Autonomous Vehicles: Intent/Slot Recognition Utilizing Audio-Visual Data

Eda Okur, Shachi H Kumar, Saurav Sahay, Lama Nachman

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title)；分类 cs.CV、cs.CL、cs.MM

Comments Presented as a short-paper at the 23rd Workshop on the Semantics and Pragmatics of Dialogue (SemDial 2019 - LondonLogue), Sep 4-6, 2019, London, UK

Journal ref Proceedings of the 23rd Workshop on the Semantics and Pragmatics of Dialogue (SEMDIAL), pp. 213-215, London, United Kingdom, September 2019

Multimodal Transfer Deep Learning with Applications in Audio-Visual Recognition

Seungwhan Moon, Suyoun Kim, Haohan Wang

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(title,abstract)

Comments 6 pages, MMML workshop at NIPS 2015

OmniFysics: Towards Physical Intelligence Evolution via Omni-Modal Signal Processing and Network Optimization

OmniFysics：通过多模态信号处理和网络优化实现物理智能进化

Minghao Han, Dingkang Yang, Yue Jiang, Yizhou Liu, Lihua Zhang

机构 * College of Intelligent Robotics and Advanced Manufacturing, Fudan University（复旦大学智能机器人与先进制造学院） ； Fysics AI 纠错

专题命中 音频语音多模态 ：omni-modal(title,abstract)；multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)

AI总结 OmniFysics通过统一图像、音频、视频和文本的多模态信号处理，结合动态物理数据引擎和物理知识注入，实现网络化AI系统的自主优化，提升物理理解能力。

Comments This work has been submitted to the IEEE for possible publication

EgoAVU: Egocentric Audio-Visual Understanding

EgoAVU：第一人称音频视觉理解

Ashish Seth, Xinhao Mei, Changsheng Zhao, Varun Nagaraja, Ernie Chang, Gregory P. Meyer, Gael Le Lan, Yunyang Xiong, Vikas Chandra, Yangyang Shi, Dinesh Manocha, Zhipeng Cai

机构 * Meta ； University of Maryland, College Park（马里兰大学College Park分校） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；multi-modal(abstract)；cross-modal(abstract)

AI总结 EgoAVU通过生成第一人称音频视觉叙述和问题，提升多模态大语言模型在第一人称视频理解中的性能。

Audit After Segmentation: Reference-Free Mask Quality Assessment for Language-Referred Audio-Visual Segmentation

分段后审计：用于语言指代音频视频分段的无参考掩码质量评估

Jinxing Zhou, Yanghao Zhou, Yaoting Wang, Zongyan Han, Jiaqi Ma, Henghui Ding, Rao Muhammad Anwer, Hisham Cholakkal

机构 * MBZUAI ； National University of Singapore（国立新加坡大学） ； Fudan University（复旦大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI、cs.MM

AI总结 本研究提出 MQA-RefAVS 任务，通过多模态大语言模型评估语言指代音频视频分段中掩码质量，提升分割准确性与可解释性。