多模态大模型

Multimodal Emotion Recognition in Conversations: A Survey of Methods, Trends, Challenges and Prospects

Chengyan Wu, Yiqiang Cai, Yang Liu, Pengxu Zhu, Yun Xue, Ziwei Gong, Julia Hirschberg, Bolei Ma

机构 * Guangdong Provincial Key Laboratory of Quantum Engineering and Quantum Materials（广东省量子工程与量子材料重点实验室） ； School of Electronic Science and Engineering (School of Microelectronics), South China Normal University（华南师范大学电子科学学院（微电子学院）） ； North Carolina Central University（北卡罗来纳中央大学） ； Georgia Institute of Technology（佐治亚理工学院） ； Columbia University（哥伦比亚大学） ； LMU Munich & Munich Center for Machine Learning（慕尼黑大学及慕尼黑机器学习中心） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments EMNLP 2025 Findings

Multimodal Fine-grained Context Interaction Graph Modeling for Conversational Speech Synthesis

Zhenqi Jia, Rui Liu, Berrak Sisman, Haizhou Li

机构 * Inner Mongolia University（内蒙古大学） ； Center for Language and Speech Processing (CLSP)（语言与语音处理中心） ； School of Artificial Intelligence, The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）人工智能学院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments Accepted by EMNLP 2025

MEAN-RIR: Multi-Modal Environment-Aware Network for Robust Room Impulse Response Estimation

Jiajian Chen, Jiakang Chen, Hang Chen, Qing Wang, Yu Gao, Jun Du

机构 * University of Science and Technology of China（科学技术大学） ； AI Research Center, Midea Group (Shanghai) Co.,Ltd.（美的集团（上海）有限公司人工智能研究中心） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 eess.AS

Comments Accepted by ASRU 2025

Spoken in Jest, Detected in Earnest: A Systematic Review of Sarcasm Recognition -- Multimodal Fusion, Challenges, and Future Prospects

Xiyuan Gao, Shekhar Nayak, Matt Coler

机构 * Campus Fryslân, University of Groningen（格罗宁根大学弗里桑校区） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments 20 pages, 7 figures, Submitted to IEEE Transactions on Affective Computing

PianoBind: A Multimodal Joint Embedding Model for Pop-piano Music

Hayeon Bang, Eunjin Choi, Seungheon Doh, Juhan Nam

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM

Comments Accepted for publication at the 26th International Society for Music Information Retrieval Conference (ISMIR 2025)

Adversarial Attacks in Multimodal Systems: A Practitioner's Survey

Shashank Kapoor, Sanjay Surendranath Girija, Lakshit Arora, Dipen Pradhan, Ankit Shetgaonkar, Aman Raj

机构 * Google（谷歌） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Comments Accepted in IEEE COMPSAC 2025

Journal ref 2025 IEEE 49th Annual Computers, Software, and Applications Conference (COMPSAC)

MoTAS: MoE-Guided Feature Selection from TTS-Augmented Speech for Enhanced Multimodal Alzheimer's Early Screening

Yongqi Shao, Binxin Mei, Cong Tan, Hong Huo, Tao Fang

机构 * Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM

Spherical Vision Transformers for Audio-Visual Saliency Prediction in 360-Degree Videos

Mert Cokelek, Halit Ozsoy, Nevrez Imamoglu, Cagri Ozcinar, Inci Ayhan, Erkut Erdem, Aykut Erdem

机构 * Department of Computer Science and Engineering, Koç University（计算机科学与工程系，科克大学） ； Department of Psychology, Boğaziçi University（心理学系，博多伊大学） ； National Institute of Advanced Industrial Science and Technology (AIST), Intelligent Platforms Research Institute（国家先进工业科学与技术研究院（AIST），智能平台研究机构） ； Department of Psychology, Boğaziçi University University（心理学系，博多伊大学） ； Department of Computer Engineering, Hacettepe University（计算机工程系，哈切塞特佩大学） ； KUIS AI Center（KUIS人工智能中心） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Comments Accepted for publication in IEEE Transaction on Pattern Analysis and Machine Intelligence (IEEE TPAMI)

Audio-Visual Feature Synchronization for Robust Speech Enhancement in Hearing Aids

Nasir Saleem, Mandar Gogate, Kia Dashtipour, Adeel Hussain, Usman Anwar, Adewale Adetomi, Tughrul Arslan, Amir Hussain

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Comments Preprint of the paper presented at Euronoise 2025 Malaga, Spain

Multimodal Appearance based Gaze-Controlled Virtual Keyboard with Synchronous Asynchronous Interaction for Low-Resource Settings

Yogesh Kumar Meena, Manish Salvi

机构 * Human-AI Interaction (HAIx) Lab, IIT Gandhinagar（人机交互（HAIx）实验室，印度加尔各答理工学院） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Take That for Me: Multimodal Exophora Resolution with Interactive Questioning for Ambiguous Out-of-View Instructions

Akira Oyama, Shoichi Hasegawa, Akira Taniguchi, Yoshinobu Hagiwara, Tadahiro Taniguchi

机构 * Ritsumeikan University（立命馆大学） ； Soka University（早稻田大学） ； Kyoto University（京都大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Comments See website at https://emergentsystemlabstudent.github.io/MIEL/. Accepted at IEEE RO-MAN 2025

Arabic Multimodal Machine Learning: Datasets, Applications, Approaches, and Challenges

Abdelhamid Haouhat, Slimane Bellaouar, Attia Nehar, Hadda Cherroun, Ahmed Abdelali

机构 * Ziane Achour University（赞赞·阿赫尔大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Mitigating Category Imbalance: Fosafer System for the Multimodal Emotion and Intent Joint Understanding Challenge

Honghong Wang, Yankai Wang, Dejun Zhang, Jing Deng, Rong Zheng

机构 * Beijing Fosafer Information Technology Co., Ltd.（北京福萨弗信息科技有限公司） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 eess.AS

Comments 2 pages. pubilshed by ICASSP2025

SpeechForensics: Audio-Visual Speech Representation Learning for Face Forgery Detection

Yachao Liang, Min Yu, Gang Li, Jianguo Jiang, Boquan Li, Feng Yu, Ning Zhang, Xiang Meng, Weiqing Huang

机构 * Institute of Information Engineering（信息工程研究所） ； Chinese Academy of Sciences（中国科学院） ； School of Cyber Security University of Chinese Academy of Sciences（中国科学院网络安全学院） ； Deakin University（德肯大学） ； Harbin Engineering University（哈尔滨工程大学） ； Institute of Computing Technology Chinese Academy of Sciences（中国科学院计算技术研究所） ； Institute of Forensic Science Ministry of Public Security（公安部刑事科学技术研究所） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Comments Accepted by NeurIPS 2024

Journal ref Advances in Neural Information Processing Systems, Volume 37, Pages 86124-86144, Year 2024

$\text{M}^3\text{PDB}$: A Multimodal, Multi-Label, Multilingual Prompt Database for Speech Generation

Boyu Zhu, Cheng Gong, Muyang Wu, Ruihao Jing, Fan Liu, Xiaolei Zhang, Chi Zhang, Xuelong Li

机构 * Institute of Artificial Intelligence (TeleAl), China Telecom（人工智能研究院（TeleAl），中国电信） ； School of Marine Science and Technology, Northwestern Polytechnical University（海洋科学与技术学院，西北工业大学） 纠错

专题命中 音频语音多模态 ：multimodal(title)；multi-modal(abstract)；分类 eess.AS

Multi-modal Adaptive Mixture of Experts for Cold-start Recommendation

Van-Khang Nguyen, Duc-Hoang Pham, Huy-Son Nguyen, Cam-Van Thi Nguyen, Hoang-Quynh Le, Duc-Trong Le

机构 * VNU University of Engineering and Technology（越南工程大学） ； Delft University of Technology（代尔夫特理工大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title)；multimodal(abstract)；分类 cs.AI

Engagement Prediction of Short Videos with Large Multimodal Models

Wei Sun, Linhan Cao, Yuqin Cao, Weixia Zhang, Wen Wen, Kaiwei Zhang, Zijian Chen, Fangfang Lu, Xiongkuo Min, Guangtao Zhai

机构 * East China Normal University（华东师范大学） ； Shanghai Jiao Tong University（上海交通大学） ； City University of Hong Kong（香港城市大学） ； Shanghai University of Electric Power（上海电力大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV

Comments The proposed method achieves first place in the ICCV VQualA 2025 EVQA-SnapUGC Challenge on short-form video engagement prediction

AV-SSAN: Audio-Visual Selective DoA Estimation through Explicit Multi-Band Semantic-Spatial Alignment

Yu Chen, Hongxu Zhu, Jiadong Wang, Kainan Chen, Xinyuan Qian

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Comments 9 pages

Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations

T. Aleksandra Ma, Sile Yin, Li-Chia Yang, Shuo Zhang

机构 * School of Music（音乐学院） ； Research（研究） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Comments Accepted into Interspeech 2025; corrected author name typo

Unraveling Hidden Representations: A Multi-Modal Layer Analysis for Better Synthetic Content Forensics

Tom Or, Omri Azencot

机构 * Ben Gurion University of the Negev（本· Gurion 内盖夫大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.AI

User Experience Estimation in Human-Robot Interaction Via Multi-Instance Learning of Multimodal Social Signals

Ryo Miyoshi, Yuki Okafuji, Takuya Iwamoto, Junya Nakanishi, Jun Baba

机构 * CyberAgent ； Osaka University（大阪大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV

Comments This paper has been accepted for presentation at IEEE/RSJ International Conference on Intelligent Robots and Systems 2025 (IROS 2025)

Multi-Input Multi-Output Target-Speaker Voice Activity Detection For Unified, Flexible, and Robust Audio-Visual Speaker Diarization

Ming Cheng, Ming Li

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Comments Accepted by IEEE Transactions on Audio, Speech, and Language Processing

AV-Deepfake1M++: A Large-Scale Audio-Visual Deepfake Benchmark with Real-World Perturbations

Zhixi Cai, Kartik Kuckreja, Shreya Ghosh, Akanksha Chuchra, Muhammad Haris Khan, Usman Tariq, Tom Gedeon, Abhinav Dhall

机构 * Monash University（墨尔本大学） ； Curtin University（Curtin大学） ； American University of Sharjah（沙迦美国大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

DFR: A Decompose-Fuse-Reconstruct Framework for Multi-Modal Few-Shot Segmentation

Shuai Chen, Fanman Meng, Xiwei Zhang, Haoran Wei, Chenhao Wu, Qingbo Wu, Hongliang Li

机构 * School of Information and Communication Engineering（信息与通信工程学院） ； University of Electronic Science and Technology of China（电子科学与技术大学） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CV

Comments 3 figures

AV-Reasoner: Improving and Benchmarking Clue-Grounded Audio-Visual Counting for MLLMs

Lidong Lu, Guo Chen, Zhiqi Li, Yicheng Liu, Tong Lu

机构 * Nanjing University（南京大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title)；multimodal(abstract)；分类 cs.CV

Comments 21 pages, 11 figures

NavVI: A Telerobotic Simulation with Multimodal Feedback for Visually Impaired Navigation in Warehouse Environments

Maisha Maimuna, Minhaz Bin Farukee, Sama Nikanfar, Mahfuza Siddiqua, Ayon Roy, Fillia Makedon

机构 * Department of Computer Science and Engineering（计算机科学与工程系） ； The University of Texas at Arlington（德克萨斯理工大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Mixture of LoRA Experts with Multi-Modal and Multi-Granularity LLM Generative Error Correction for Accented Speech Recognition

Bingshen Mu, Kun Wei, Pengcheng Guo, Lei Xie

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 eess.AS

Comments IEEE Transactions on Audio, Speech and Language Processing

AVFSNet: Audio-Visual Speech Separation for Flexible Number of Speakers with Multi-Scale and Multi-Task Learning

Daning Zhang, Ying Wei

机构 * School of Control Science and Engineering, Shandong University（控制科学与工程学院，山东大学） ； Industrial Technology Research Institte of Shandong Province（山东省工业技术研究院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Training Strategies for Modality Dropout Resilient Multi-Modal Target Speaker Extraction

Srikanth Korse, Mohamed Elminshawi, Emanuel A. P. Habets, Srikanth Raj Chetupalli

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 eess.AS

Comments Published in ICASSPW 2024 (HSCMA)

Multimodal Lyrics-Rhythm Matching

Callie C. Liao, Duoduo Liao, Jesse Guessford

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 eess.AS

Comments Accepted by 2022 IEEE International Conference on Big Data (IEEE Big Data 2022)

Journal ref IEEE BigData, Year: 2022; Pages: 3622-3630