多模态大模型 - arXivDaily 专题

2409.09545 2025-07-30 cs.SD cs.LG eess.AS 85%

Multi-Microphone and Multi-Modal Emotion Recognition in Reverberant Environment

Ohad Cohen, Gershon Hazan, Sharon Gannot

机构 * Faculty of Engineering Bar-Ilan University Ramat-Gan, Israel（工程学院巴伊兰大学拉马特-甘, 以色列）

专题命中音频语音多模态：multi-modal(title,abstract)；multimodal(abstract)；audio-visual(abstract)；分类 eess.AS

Comments 5 pages, 4 figures, 2 tables. Accepted to EUSIPCO 2025

2310.13103 2025-07-08 cs.CV cs.AI cs.LG cs.MM cs.SD eess.AS 85%

AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection

Ammarah Hashmi, Sahibzada Adil Shahzad, Chia-Wen Lin, Yu Tsao, Hsin-Min Wang

机构 * Social Networks and Human-Centered Computing Program, Taiwan International Graduate Program, Academia Sinica（社会网络与以人为本计算计划、台湾国际研究生计划、中国科学院）； Institute of Information Systems and Applications, National Tsing Hua University（信息系统与应用研究所、国立清华大学）； Department of Computer Science, National Chengchi University（计算机科学系、国立成功大学）； Research Center for Information Technology Innovation, Academia Sinica（信息技术创新研究中心、中国科学院）； Institute of Information Science, Academia Sinica（信息科学研究所、中国科学院）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

2506.05414 2025-06-09 cs.CV cs.AI cs.LG cs.MM cs.SD eess.AS 85%

SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing

Mingfei Chen, Zijun Cui, Xiulong Liu, Jinlin Xiang, Caleb Zheng, Jingyuan Li, Eli Shlizerman

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Project website with demo videos: https://zijuncui02.github.io/SAVVY/

2502.04328 2025-06-04 cs.CV cs.CL cs.MM cs.SD eess.AS eess.IV 85%

Ola: Pushing the Frontiers of Omni-Modal Language Model

Zuyan Liu, Yuhao Dong, Jiahui Wang, Ziwei Liu, Winston Hu, Jiwen Lu, Yongming Rao

机构 * Tsinghua University（清华大学）； Tencent Hunyuan Research（腾讯文睿实验室）； S-Lab, NTU（国立科技大学S实验室）

专题命中音频语音多模态：omni-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

2504.04495 2025-06-03 cs.CV 85%

AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection

Peng Wu, Wanshun Su, Guansong Pang, Yujia Sun, Qingsen Yan, Peng Wang, Yanning Zhang

机构 * School of Computer Science, Northwestern Polytechnical University（西北工业大学计算机科学学院）； School of Computing and Information Systems, Singapore Management University（新加坡管理学院计算与信息系统学院）； School of Artifical Intelligence, Xidian University（西安电子科技大学人工智能学院）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments 12 pages, 6 figures, 9 tables. This work has been submitted to the IEEE for possible publication

2504.14267 2025-04-22 cs.CV 85%

Text-Audio-Visual-conditioned Diffusion Model for Video Saliency Prediction

Li Yu, Xuanzhe Sun, Wei Zhou, Moncef Gabbouj

机构 * School of Computer Science, Nanjing University of Information Science and Technology（信息科学与技术南京大学计算机科学学院）； Jiangsu Collaborative Innovation Center of Atmospheric Environment and Equipment Technology, Nanjing University of Information Science and Technology（大气环境与设备技术江苏省协同创新中心）； School of Computer Science and Informatics, Cardiff University（计算机科学与信息学院卡斯尔大学）； Faculty of Information Technology and Communication Sciences, Tampere University（信息科技与通讯科学学院坦佩雷大学）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；image-text(abstract)；分类 cs.CV

2504.10070 2025-04-17 cs.CV 85%

DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction

Kiana Hooshanfar, Alireza Hosseini, Ahmad Kalhor, Babak Nadjar Araabi

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

2406.06965 2025-04-04 cs.CV 85%

Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges

Ping Liu, Qiqi Tao, Joey Tianyi Zhou

专题命中音频语音多模态：multi-modal(title,abstract)；multimodal(abstract)；audio-visual(abstract)；分类 cs.CV

Comments P. Liu is with the Department of Computer Science and Engineering, University of Nevada, Reno, NV, 89512. Q. Tao and J. Zhou are with Centre for Frontier AI Research (CFAR), and Institute of High Performance Computing (IHPC), A*STAR, Singapore. J. Zhou is also with Centre for Advanced Technologies in Online Safety (CATOS), A*STAR, Singapore. J. Zhou is the corresponding author

2504.01049 2025-04-03 cs.CV cs.LG 85%

SViQA: A Unified Speech-Vision Multimodal Model for Textless Visual Question Answering

Bingxin Li

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV

2504.00561 2025-04-02 cs.CV 85%

Continual Cross-Modal Generalization

Yan Xia, Hai Huang, Minghui Fang, Zhou Zhao

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；image-text(abstract)；分类 cs.CV

2502.05766 2025-02-11 eess.AS cs.SD 85%

Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models

Jing-Xuan Zhang, Genshun Wan, Jianqing Gao, Zhen-Hua Ling

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments accepted to Pattern Recognition

2501.09608 2025-01-17 cs.SD cs.AI cs.CV cs.IR cs.MM eess.AS 85%

Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning

Donghuo Zeng, Kazushi Ikeda

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 5 pages, 3 figures, 2 tables. Accepted by ICASSP 2025

2501.09352 2025-01-17 cs.LG cs.MM eess.IV 85%

PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning

Xianghu Yue, Yiming Chen, Xueyi Zhang, Xiaoxue Gao, Mengling Feng, Mingrui Lao, Huiping Zhuang, Haizhou Li

专题命中音频语音多模态：multi-modal(title,abstract)；image-text(abstract)；audio-visual(abstract)；分类 cs.MM

2412.05558 2024-12-10 cs.SD cs.AI cs.CV cs.MM eess.AS 85%

WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition

Feng Li, Jiusong Luo, Wanjun Xia

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by 31st International Conference on MultiMedia Modeling (MMM2025)

2410.23230 2024-11-01 cs.CV cs.AI cs.LG cs.MM cs.SD eess.AS 85%

Aligning Audio-Visual Joint Representations with an Agentic Workflow

Shentong Mo, Yibing Song

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

2408.01952 2024-08-06 cs.CV 85%

CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization

Xiang He, Xiangxi Liu, Yang Li, Dongcheng Zhao, Guobin Shen, Qingqun Kong, Xin Yang, Yi Zeng

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ACM MM 2024. Code is available at this https://github.com/Brain-Cog-Lab/CACE-Net

2406.15160 2024-06-24 eess.AS eess.SP 85%

Exploring Audio-Visual Information Fusion for Sound Event Localization and Detection In Low-Resource Realistic Scenarios

Ya Jiang, Qing Wang, Jun Du, Maocheng Hu, Pengfei Hu, Zeyan Liu, Shi Cheng, Zhaoxu Nian, Yuxuan Dong, Mingqi Cai, Xin Fang, Chin-Hui Lee

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 eess.AS

Comments accepted by icme2024

2406.08802 2024-06-14 eess.AS cs.SD 85%

DubWise: Video-Guided Speech Duration Control in Multimodal LLM-based Text-to-Speech for Dubbing

Neha Sahipjohn, Ashishkumar Gudmalwar, Nirmesh Shah, Pankaj Wasnik, Rajiv Ratn Shah

专题命中音频语音多模态：multimodal(title)；multi-modal(abstract)；cross-modal(abstract)；audio-visual(abstract)

Comments Accepted at INTERSPEECH 2024

2312.02512 2024-03-27 cs.CV cs.AI cs.MM eess.AS 85%

AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation

Jeongsoo Choi, Se Jin Park, Minsu Kim, Yong Man Ro

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments CVPR 2024. Code & Demo: https://choijeongsoo.github.io/av2av

2403.14974 2024-03-25 cs.CV 85%

AVT2-DWF: Improving Deepfake Detection with Audio-Visual Fusion and Dynamic Weighting Strategies

Rui Wang, Dengpan Ye, Long Tang, Yunming Zhang, Jiacheng Deng

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

2403.06679 2024-03-12 cs.CV 85%

Answering Diverse Questions via Text Attached with Key Audio-Visual Clues

Qilang Ye, Zitong Yu, Xin Liu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

2305.12397 2023-12-11 cs.CV 85%

Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios

Yuanyuan Jiang, Jianqin Yin

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted to EMNLP 2023 Findings

2309.10724 2023-09-20 cs.CV cs.AI cs.MM cs.SD eess.AS 85%

Sound Source Localization is All about Cross-Modal Alignment

Arda Senocak, Hyeonggon Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, Joon Son Chung

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments ICCV 2023

2309.03869 2023-09-08 cs.CV 85%

Text-to-feature diffusion for audio-visual few-shot learning

Otniel-Bogdan Mercea, Thomas Hummel, A. Sophia Koepke, Zeynep Akata

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments DAGM GCPR 2023

2206.10421 2022-07-12 cs.SD cs.AI cs.CV cs.MM eess.AS 85%

Rethinking Audio-visual Synchronization for Active Speaker Detection

Abudukelimu Wuerkaixi, You Zhang, Zhiyao Duan, Changshui Zhang

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by IEEE International Workshop on Machine Learning for Signal Processing (MLSP 2022)

2110.13412 2021-10-27 cs.CV 85%

TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation

Tanzila Rahman, Mengyu Yang, Leonid Sigal

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

Comments 10 pages, 5 Figures, Neurips 2021

Journal ref https://nips.cc/Conferences/2021

2006.09199 2021-07-01 cs.CV cs.CL cs.MM cs.SD eess.AS 85%

AVLnet: Learning Audio-Visual Language Representations from Instructional Videos

Andrew Rouditchenko, Angie Boggust, David Harwath, Brian Chen, Dhiraj Joshi, Samuel Thomas, Kartik Audhkhasi, Hilde Kuehne, Rameswar Panda, Rogerio Feris, Brian Kingsbury, Michael Picheny, Antonio Torralba, James Glass

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments A version of this work has been accepted to Interspeech 2021

2004.14858 2020-07-10 cs.MM cs.CL cs.CV cs.SD eess.AS 85%

MuSe 2020 -- The First International Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop

Lukas Stappen, Alice Baird, Georgios Rizos, Panagiotis Tzirakis, Xinchen Du, Felix Hafner, Lea Schumann, Adria Mallol-Ragolta, Björn W. Schuller, Iulia Lefter, Erik Cambria, Ioannis Kompatsiaris

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Baseline Paper MuSe 2020, MuSe Workshop Challenge, ACM Multimedia

1908.07094 2019-08-21 cs.CV 85%

Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck

Shuang Ma, Daniel McDuff, Yale Song

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments ICCV 2019

1807.09991 2018-07-27 cs.AI cs.RO 85%

Multi-modal Feedback for Affordance-driven Interactive Reinforcement Learning

Francisco Cruz, German I. Parisi, Stefan Wermter

专题命中音频语音多模态：multi-modal(title,abstract)；multimodal(abstract)；audio-visual(abstract)；分类 cs.AI

Comments Accepted at IEEE IJCNN 2018, Rio de Janeiro, Brazil

1. 音频语音多模态 4549 篇

Multi-Microphone and Multi-Modal Emotion Recognition in Reverberant Environment

AVTENet: A Human-Cognition-Inspired Audio-Visual Transformer-Based Ensemble Network for Video Deepfake Detection

SAVVY: Spatial Awareness via Audio-Visual LLMs through Seeing and Hearing

Ola: Pushing the Frontiers of Omni-Modal Language Model

AVadCLIP: Audio-Visual Collaboration for Robust Video Anomaly Detection

Text-Audio-Visual-conditioned Diffusion Model for Video Saliency Prediction

DTFSal: Audio-Visual Dynamic Token Fusion for Video Saliency Prediction

Evolving from Single-modal to Multi-modal Facial Deepfake Detection: Progress and Challenges

SViQA: A Unified Speech-Vision Multimodal Model for Textless Visual Question Answering

Continual Cross-Modal Generalization

Audio-Visual Representation Learning via Knowledge Distillation from Speech Foundation Models

Metric Learning with Progressive Self-Distillation for Audio-Visual Embedding Learning

PAL: Prompting Analytic Learning with Missing Modality for Multi-Modal Class-Incremental Learning

WavFusion: Towards wav2vec 2.0 Multimodal Speech Emotion Recognition

Aligning Audio-Visual Joint Representations with an Agentic Workflow

CACE-Net: Co-guidance Attention and Contrastive Enhancement for Effective Audio-Visual Event Localization

Exploring Audio-Visual Information Fusion for Sound Event Localization and Detection In Low-Resource Realistic Scenarios

DubWise: Video-Guided Speech Duration Control in Multimodal LLM-based Text-to-Speech for Dubbing

AV2AV: Direct Audio-Visual Speech to Audio-Visual Speech Translation with Unified Audio-Visual Speech Representation

AVT2-DWF: Improving Deepfake Detection with Audio-Visual Fusion and Dynamic Weighting Strategies

Answering Diverse Questions via Text Attached with Key Audio-Visual Clues

Target-Aware Spatio-Temporal Reasoning via Answering Questions in Dynamics Audio-Visual Scenarios

Sound Source Localization is All about Cross-Modal Alignment

Text-to-feature diffusion for audio-visual few-shot learning

Rethinking Audio-visual Synchronization for Active Speaker Detection

TriBERT: Full-body Human-centric Audio-visual Representation Learning for Visual Sound Separation

AVLnet: Learning Audio-Visual Language Representations from Instructional Videos

MuSe 2020 -- The First International Multimodal Sentiment Analysis in Real-life Media Challenge and Workshop

Unpaired Image-to-Speech Synthesis with Multimodal Information Bottleneck

Multi-modal Feedback for Affordance-driven Interactive Reinforcement Learning