多模态大模型 - arXivDaily 专题

2510.19358 2025-10-23 cs.CL cs.AI 84%

M3-SLU: Evaluating Speaker-Attributed Reasoning in Multimodal Large Language Models

Yejin Kwon, Taewoo Kang, Hyunsoo Yoon, Changouk Kim

专题命中音频语音多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

Comments Submitted to LREC 2026. 11 pages, 5 figures

2403.16276 2025-10-09 cs.CV cs.AI 84%

Empowering LLMs with Pseudo-Untrimmed Videos for Audio-Visual Temporal Understanding

Yolo Yunlong Tang, Daiki Shimada, Jing Bi, Mingqian Feng, Hang Hua, Chenliang Xu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted to AAAI 2025

2501.04686 2025-10-07 cs.CL cs.AI cs.LG 84%

Unlocking Multimodal Mathematical Reasoning via Process Reward Model

Ruilin Luo, Zhuofan Zheng, Yifan Wang, Xinzhe Ni, Zicheng Lin, Songtao Jiang, Yiyao Yu, Chufan Shi, Lei Wang, Ruihang Chu, Jin Zeng, Yujiu Yang

机构 * Tsinghua University（清华大学）； ByteDance（字节跳动）； Zhejiang University（浙江大学）； Ping An Technology (Shenzhen) Co., Ltd（平安科技（深圳）有限公司）

专题命中音频语音多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

Comments NeurIPS 2025 Main Track

2509.25652 2025-10-01 cs.AI cs.MM cs.SD 84%

Iterative Residual Cross-Attention Mechanism: An Integrated Approach for Audio-Visual Navigation Tasks

Hailong Zhang, Yinfeng Yu, Liejun Wang, Fuchun Sun, Wendong Zheng

机构 * Xinjiang Multimodal Intelligent Processing and Information Security Engineering Technology Research Center, School of Computer Science and Technology, Xinjiang University（新疆多模态智能处理与信息安全工程技术创新中心，计算机科学与技术学院，新疆大学）； Department of Computer Science and Technology, Tsinghua University（计算机科学与技术系，清华大学）； School of Electrical Engineering and Automation, Tianjin University of Technology（电气工程与自动化学院，天津工业大学）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM

Comments Accepted for publication by IEEE International Conference on Systems, Man, and Cybernetics 2025

2505.15772 2025-10-01 cs.SD cs.CL eess.AS 84%

MIKU-PAL: An Automated and Standardized Multi-Modal Method for Speech Paralinguistic and Affect Labeling

Yifan Cheng, Ruoyi Zhang, Jiatong Shi

机构 * Santa Clara, CA, USA（美国圣克拉拉）； Carnegie Mellon University（卡内基梅隆大学）； Huazhong University of Science and Technology（华中科技大学）； Nanjing University of Information Science and Technology（南京信息工程大学）

专题命中音频语音多模态：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CL、eess.AS

Comments Accepted by Interspeech

Journal ref Proc. of Interspeech2025

2509.00723 2025-09-03 cs.AI cs.MM 84%

OmniDPO: A Preference Optimization Framework to Address Omni-Modal Hallucination

Junzhe Chen, Tianshu Zhang, Shiyu Huang, Yuwei Niu, Chao Sun, Rongzhou Zhang, Guanyu Zhou, Lijie Wen, Xuming Hu

机构 * Tsinghua University（清华大学）； The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州））； OpenRL ； Chongqing University（重庆大学）

专题命中音频语音多模态：omni-modal(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM

2508.20546 2025-08-29 cs.MM cs.AI 84%

MM-HSD: Multi-Modal Hate Speech Detection in Videos

Berta Céspedes-Sarrias, Carlos Collado-Capell, Pablo Rodenas-Ruiz, Olena Hrynenko, Andrea Cavallaro

机构 * EPFL（苏黎世联邦理工学院）； Idiap Research Institute（日内瓦研究所）

专题命中音频语音多模态：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI、cs.MM

Comments Accepted at ACM Multimedia 2025

2408.16564 2025-08-28 cs.MM cs.SD eess.AS 84%

Human-Inspired Computing for Robust and Efficient Audio-Visual Speech Recognition

Qianhui Liu, Jiadong Wang, Yang Wang, Xin Yang, Gang Pan, Haizhou Li

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM、eess.AS

Comments aceepted by IEEE TC

2504.11002 2025-08-13 cs.SD cs.MM eess.AS 84%

Dopamine Audiobook: A Training-free MLLM Agent for Emotional and Immersive Audiobook Generation

Yan Rong, Shan Yang, Chenxing Li, Dong Yu, Li Liu

专题命中音频语音多模态：MLLM(title,abstract)；multimodal(abstract)；分类 cs.MM、eess.AS

2508.00632 2025-08-04 cs.AI cs.MA cs.MM 84%

Multi-Agent Game Generation and Evaluation via Audio-Visual Recordings

Alexia Jolicoeur-Martineau

机构 * Samsung SAIL Montréal（三星SAIL蒙特利尔）

专题命中音频语音多模态：audio-visual(title,abstract)；omni-modal(abstract)；分类 cs.AI、cs.MM

2507.21588 2025-07-30 cs.AI cs.CV 84%

Progressive Homeostatic and Plastic Prompt Tuning for Audio-Visual Multi-Task Incremental Learning

Jiong Yin, Liang Li, Jiehua Zhang, Yuhan Gao, Chenggang Yan, Xichun Sheng

机构 * Hangzhou Dianzi University（杭州电子科技大学）； Institute of Computing Technology, Chinese Academy of Sciences（中国科学院计算技术研究所）； Xi’an Jiaotong University（西安交通大学）； Macao Polytechnic University（澳门 polytechnic university）

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICCV 2025

2507.07015 2025-07-10 cs.CV cs.LG cs.MM 84%

MST-Distill: Mixture of Specialized Teachers for Cross-Modal Knowledge Distillation

Hui Li, Pengfei Yang, Juanyang Chen, Le Dong, Yanxin Chen, Quan Wang

机构 * Xidian University（西安电子科技大学）

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Comments Accepted to ACM MM 2025 (The 33rd ACM International Conference on Multimedia)

2501.16780 2025-07-08 cs.SD cs.HC cs.MM eess.AS 84%

AVE Speech: A Comprehensive Multi-Modal Dataset for Speech Recognition Integrating Audio, Visual, and Electromyographic Signals

Dongliang Zhou, Yakun Zhang, Jinghan Wu, Xingyu Zhang, Liang Xie, Erwei Yin

机构 * Defense Innovation Institute, Academy of Military Sciences, Beijing, China（国防科技研究院，军事科学院，北京，中国）； Tianjin Artificial Intelligence Innovation Center, Tianjin, China（天津人工智能创新中心，天津，中国）； Harbin Institute of Technology, Shenzhen, China（哈尔滨工业大学，深圳，中国）； Tianjin University, Tianjin, China（天津大学，天津，中国）

专题命中音频语音多模态：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

Comments The paper has been accepted by IEEE Transactions on Human-Machine Systems

2507.00055 2025-07-02 cs.LG cs.HC cs.MM eess.AS eess.IV eess.SP 84%

Leveraging Unlabeled Audio-Visual Data in Speech Emotion Recognition using Knowledge Distillation

Varsha Pendyala, Pedro Morgado, William Sethares

机构 * Department of Electrical and Computer Engineering（电气与计算机工程系）

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.MM、eess.AS

Comments Accepted at INTERSPEECH 2025

2506.05899 2025-06-09 cs.SD cs.AI eess.AS 84%

WhisQ: Cross-Modal Representation Learning for Text-to-Music MOS Prediction

Jakaria Islam Emon, Kazi Tamanna Alam, Md. Abu Salek

机构 * Hokkaido Denshikiki Co., Ltd.（Hokkaido Denshikiki公司）

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.AI、eess.AS

Comments 3 pages

2408.03505 2025-06-03 cs.CL cs.AI cs.DC 84%

Optimus: Accelerating Large-Scale Multi-Modal LLM Training by Bubble Exploitation

Weiqi Feng, Yangrui Chen, Shaoyu Wang, Yanghua Peng, Haibin Lin, Minlan Yu

机构 * Harvard University（哈佛大学）； Bytedance（字节跳动）； University of Southern California（南加州大学）

专题命中音频语音多模态：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CL、cs.AI

2506.01111 2025-06-03 cs.SD cs.AI eess.AS 84%

FusionAudio-1.2M: Towards Fine-grained Audio Captioning with Multimodal Contextual Fusion

Shunian Chen, Xinyuan Xie, Zheshu Chen, Liyan Zhao, Owen Lee, Zhan Su, Qilin Sun, Benyou Wang

机构 * The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳））

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI、eess.AS

2505.18972 2025-05-27 eess.AS cs.AI 84%

Revival with Voice: Multi-modal Controllable Text-to-Speech Synthesis

Minsu Kim, Pingchuan Ma, Honglie Chen, Stavros Petridis, Maja Pantic

机构 * Meta AIUK（Meta AI英国分公司）； Imperial College London（伦敦帝国学院）

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.AI、eess.AS

Comments Interspeech 2025

2505.03186 2025-05-16 cs.SD cs.CV eess.AS 84%

CoGenAV: Versatile Audio-Visual Representation Learning via Contrastive-Generative Synchronization

Detao Bai, Zhiheng Ma, Xihan Wei, Liefeng Bo

机构 * Tongyi Lab, Alibaba Group（通义实验室，阿里巴巴集团）

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、eess.AS

2505.09615 2025-05-15 cs.CV cs.SD eess.AS 84%

UWAV: Uncertainty-weighted Weakly-supervised Audio-Visual Video Parsing

Yung-Hsuan Lai, Janek Ebbers, Yu-Chiang Frank Wang, François Germain, Michael Jeffrey Jones, Moitreya Chatterjee

机构 * Graduate Institute of Communication Engineering, National Taiwan University（国立台湾大学通信工程研究所）； NVIDIA, Taiwan（台湾NVIDIA公司）； Mitsubishi Electric Research Labs (MERL)（三菱电机研究实验室）

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments CVPR 2025

2502.01547 2025-05-08 eess.AS cs.CV cs.SD 84%

mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition

Andrew Rouditchenko, Samuel Thomas, Hilde Kuehne, Rogerio Feris, James Glass

机构 * MIT（麻省理工学院）； MIT-IBM Watson AI Lab（麻省理工-IBM沃森人工智能实验室）； University of Tuebingen（图宾根大学）

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、eess.AS

Comments Accepted in Signal Processing Letters. Code at https://github.com/roudimit/whisper-flamingo

2407.06606 2025-05-07 cs.CV cs.CL 84%

Tailored Design of Audio-Visual Speech Recognition Models using Branchformers

David Gimeno-Gómez, Carlos-D. Martínez-Hinarejos

机构 * Pattern Recognition and Human Language Technology research center, Universitat Politècnica de València, Spain（模式识别与人类语言技术研究中心，西班牙巴塞罗那理工大学）

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted in Computer Speech & Language journal of Elsevier

2504.18539 2025-05-01 eess.AS cs.LG cs.MM cs.SD 84%

Multi-Task Corrupted Prediction for Learning Robust Audio-Visual Speech Representation

Sungnyun Kim, Sungwoo Cho, Sangmin Bae, Kangwook Jang, Se-Young Yun

机构 * Kim Jaechul Graduate School of AI, KAIST（金jaechul人工智能研究生院，韩国科学技术院）； School of Electrical Engineering, KAIST（电气工程学院，韩国科学技术院）

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.MM、eess.AS

Comments ICLR 2025; 22 pages, 6 figures, 14 tables

2504.15066 2025-04-22 cs.MM cs.AI 84%

Chinese-LiPS: A Chinese audio-visual speech recognition dataset with Lip-reading and Presentation Slides

Jinghua Zhao, Yuhang Jia, Shiyao Wang, Jiaming Zhou, Hui Wang, Yong Qin

机构 * College of Computer Science, Nankai University, Tianjin, China（南开大学计算机科学学院）

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM

Comments 6 pages, 7 figures

2412.15322 2025-04-09 cs.CV cs.LG cs.SD eess.AS 84%

MMAudio: Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis

Ho Kei Cheng, Masato Ishii, Akio Hayakawa, Takashi Shibuya, Alexander Schwing, Yuki Mitsufuji

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments Accepted to CVPR 2025. Project page: https://hkchengrex.github.io/MMAudio

2503.20919 2025-03-31 cs.CL cs.SD eess.AS 84%

GatedxLSTM: A Multimodal Affective Computing Approach for Emotion Recognition in Conversations

Yupei Li, Qiyang Sun, Sunil Munthumoduku Krishna Murthy, Emran Alturki, Björn W. Schuller

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、eess.AS

2503.22265 2025-03-31 cs.CV cs.SD eess.AS 84%

DeepAudio-V1:Towards Multi-Modal Multi-Stage End-to-End Video to Speech and Audio Generation

Haomin Zhang, Chang Liu, Junjie Zheng, Zihao Chen, Chaofan Ding, Xinhan Di

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

Comments 11 pages, 5 figures

2503.16585 2025-03-24 cs.CL cs.CV cs.DC cs.LG 84%

Distributed LLMs and Multimodal Large Language Models: A Survey on Advances, Challenges, and Future Directions

Hadi Amini, Md Jueal Mia, Yasaman Saadati, Ahmed Imteaj, Seyedsina Nabavirazavi, Urmish Thakker, Md Zarif Hossain, Awal Ahmed Fime, S. S. Iyengar

专题命中音频语音多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

2411.11278 2025-03-12 cs.CV cs.MM 84%

Towards Open-Vocabulary Audio-Visual Event Localization

Jinxing Zhou, Dan Guo, Ruohao Guo, Yuxin Mao, Jingjing Hu, Yiran Zhong, Xiaojun Chang, Meng Wang

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Comments accepted by CVPR 2025; Project page: https://github.com/jasongief/OV-AVEL

2503.02284 2025-03-05 cs.CV cs.AI 84%

Semi-Supervised Audio-Visual Video Action Recognition with Audio Source Localization Guided Mixup

Seokun Kang, Taehwan Kim

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

1. 音频语音多模态 4549 篇