多模态大模型

Social-MAE: A Transformer-Based Multimodal Autoencoder for Face and Voice

Hugo Bohy, Minh Tran, Kevin El Haddad, Thierry Dutoit, Mohammad Soleymani

机构 * Numediart Institute, ISIA Lab, University of Mons（Numediart研究院、ISIA实验室、蒙斯大学） ； Institute for Creative Technologies, University of Southern California（创意技术研究所、南加州大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments 5 pages, 3 figures, IEEE FG 2024 conference

Journal ref 2024 IEEE 18th International Conference on Automatic Face and Gesture Recognition (FG)

Adversarial Illusions in Multi-Modal Embeddings

Tingwei Zhang, Rishi Jha, Eugene Bagdasaryan, Vitaly Shmatikov

机构 * Cornell University（康奈尔大学） ； University of Massachusetts Amherst（马萨诸塞大学阿默斯特分校） ； Cornell Tech（康奈尔科技） 纠错

专题命中 音频语音多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments In USENIX Security'24

The Man Behind the Sound: Demystifying Audio Private Attribute Profiling via Multimodal Large Language Model Agents

Lixu Wang, Kaixiang Yao, Xinfeng Li, Dong Yang, Haoyang Li, Xiaofeng Wang, Wei Dong

机构 * Nanyang Technological University（南洋理工大学） ； The University of Tokyo（东京大学） ； Hong Kong Polytechnic University（香港理工大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 eess.AS

Comments 22 pages, 4 figures

Heuristic-Induced Multimodal Risk Distribution Jailbreak Attack for Multimodal Large Language Models

Ma Teng, Jia Xiaojun, Duan Ranjie, Li Xinfeng, Huang Yihao, Jia Xiaoshuang, Chu Zhixuan, Ren Wenqi

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments ICCV 2025

MusiXQA: Advancing Visual Music Understanding in Multimodal Large Language Models

Jian Chen, Wenye Ma, Penghang Liu, Wei Wang, Tengwei Song, Ming Li, Chenguang Wang, Jiayu Qin, Ruiyi Zhang, Changyou Chen

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments Under Review

MUG: Pseudo Labeling Augmented Audio-Visual Mamba Network for Audio-Visual Video Parsing

Langyu Wang, Bingke Zhu, Yingying Chen, Yiyuan Zhang, Ming Tang, Jinqiao Wang

机构 * Foundation Model Research Center, Institute of Automation, Chinese Academy of Sciences, China（中国科学院自动化研究所基础模型研究中心） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accpted by ICCV 2025

eMotions: A Large-Scale Dataset and Audio-Visual Fusion Network for Emotion Analysis in Short-form Videos

Xuecheng Wu, Dingkang Yang, Danlei Huang, Xinyi Yin, Yifan Wang, Jia Zhang, Jiayu Nie, Liangyu Fu, Yang Liu, Junxiao Xue, Hadi Amirpour, Wei Zhou

机构 * School of Computer Science and Technology, Xi’an Jiaotong University（计算机科学与技术学院，西安交通大学） ； College of Intelligent Robotics and Advanced Manufacturing, Fudan University & ByteDance（智能机器人与先进制造学院，复旦大学 & 字节跳动） ； School of Cyber Science and Engineering, Zhengzhou University（网络科学与工程学院，郑州大学） ； Institute of Advanced Technology, University of Science and Technology of China（先进技术研究院，中国科学技术大学） ； Inspur Electronic Information Industry Co., Ltd（Inspur电子信息产业有限公司） ； Department of Computer Science, The University of Toronto（计算机科学系，多伦多大学） ； Research Center for Space Computing System, Zhejiang Lab（空间计算系统研究中心，浙江实验室） ； Institute of Information Technology, University of Klagenfurt（信息技术学院，克雷格福特大学） ； School of Computer Science and Informatics, Cardiff University（计算机科学与信息学院，卡迪夫大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

SVC 2025: the First Multimodal Deception Detection Challenge

Xun Lin, Xiaobao Guo, Taorui Wang, Yingjie Ma, Jiajian Huang, Jiayu Zhang, Junzhe Cao, Zitong Yu

机构 * Great Bay University（大亚湾大学） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments Accepted by Workshop SVC of ACM MM 2025

From Waveforms to Pixels: A Survey on Audio-Visual Segmentation

Jia Li, Yapeng Tian

机构 * Department of Computer Science, The University of Texas at Dallas（计算机科学系，德克萨斯大学达拉斯分校） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Implicit Counterfactual Learning for Audio-Visual Segmentation

Mingfeng Zha, Tianyu Li, Guoqing Wang, Peng Wang, Yangyang Wu, Yang Yang, Heng Tao Shen

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted by ICCV 2025

What's Making That Sound Right Now? Video-centric Audio-Visual Localization

Hahyeon Choi, Junhoo Lee, Nojun Kwak

机构 * Seoul National University（首尔国立大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Published at ICCV 2025. Project page: https://hahyeon610.github.io/Video-centric_Audio_Visual_Localization/

EXPOTION: Facial Expression and Motion Control for Multimodal Music Generation

Fathinah Izzati, Xinyue Li, Gus Xia

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

The role of audio-visual integration in the time course of phonetic encoding in self-supervised speech models

Yi Wang, Oli Danyi Liu, Peter Bell

机构 * Wang Liu Bell University of Edinburgh（王刘贝尔爱丁堡大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments Accepted by Interspeech 2025

Object-aware Sound Source Localization via Audio-Visual Scene Understanding

Sung Jin Um, Dongjin Kim, Sangmin Lee, Jung Uk Kim

机构 * Kyung Hee University（庆熙大学） ； KAIST AI（韩国科学技术院人工智能研究所） ； Sungkyunkwan University（成均馆大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Accepted at CVPR 2025

Journal ref Proceedings of the Computer Vision and Pattern Recognition Conference (CVPR), 2025, pp. 8342-8351

VideoMathQA: Benchmarking Mathematical Reasoning via Multimodal Understanding in Videos

Hanoona Rasheed, Abdelrahman Shaker, Anqi Tang, Muhammad Maaz, Ming-Hsuan Yang, Salman Khan, Fahad Shahbaz Khan

机构 * MBZUAI（穆桑大学人工智能研究所） ； University of California Merced（加州大学默塞德分校） ； Google Research（谷歌研究） ； Australian National University（澳大利亚国立大学） ； Linköping University（林肯大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments VideoMathQA Technical Report

Stream-Omni: Simultaneous Multimodal Interactions with Large Language-Vision-Speech Model

Shaolei Zhang, Shoutao Guo, Qingkai Fang, Yan Zhou, Yang Feng

机构 * Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, Chinese Academy of Sciences (ICT/CAS)（中国科学院智能信息处理重点实验室） ； Key Laboratory of AI Safety, Chinese Academy of Sciences（中国科学院人工智能安全重点实验室） ； University of Chinese Academy of Sciences, Beijing, China（中国科学院大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Code: https://github.com/ictnlp/Stream-Omni , Model: https://huggingface.co/ICTNLP/stream-omni-8b

Question-Aware Gaussian Experts for Audio-Visual Question Answering

Hongyeob Kim, Inyoung Jung, Dayoon Suh, Youjia Zhang, Sangmin Lee, Sungeun Hong

机构 * Sungkyunkwan University（成均馆大学） ； Purdue University（普渡大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments CVPR 2025. Code is available at https://github.com/AIM-SKKU/QA-TIGER

Multimodal Zero-Shot Framework for Deepfake Hate Speech Detection in Low-Resource Languages

Rishabh Ranjan, Likhith Ayinala, Mayank Vatsa, Richa Singh

机构 * Indian Institute of Technology JodhpurIndia（印度理工学院贾姆普尔分校） ； Columbia UniversityUSA（哥伦比亚大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 eess.AS

Comments Accepted in Interpseech 2025

GIA-MIC: Multimodal Emotion Recognition with Gated Interactive Attention and Modality-Invariant Learning Constraints

Jiajun He, Jinyi Mi, Tomoki Toda

机构 * Nagoya University（名古屋大学） 纠错

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

Comments Accepted by INTERSPEECH 2025

SAM2-LOVE: Segment Anything Model 2 in Language-aided Audio-Visual Scenes

Yuji Wang, Haoran Xu, Yong Liu, Jiaze Li, Yansong Tang

机构 * Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments CVPR 2025

Con Instruction: Universal Jailbreaking of Multimodal Large Language Models via Non-Textual Modalities

Jiahui Geng, Thy Thy Tran, Preslav Nakov, Iryna Gurevych

专题命中 音频语音多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CL

Seeing Soundscapes: Audio-Visual Generation and Separation from Soundscapes Using Audio-Visual Separator

Minjae Kang, Martim Brandão

机构 * King’s College London（伦敦国王学院） 纠错

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Originally submitted to CVPR 2025 on 2024-11-15 with paper ID 15808

Deconfounded Reasoning for Multimodal Fake News Detection via Causal Intervention

Moyang Liu, Kaiying Yan, Yukun Liu, Ruibo Fu, Zhengqi Wen, Xuefei Liu, Chenxing Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.MM

DiMoDif: Discourse Modality-information Differentiation for Audio-visual Deepfake Detection and Localization

Christos Koutlis, Symeon Papadopoulos

专题命中 音频语音多模态 ：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

Lend a Hand: Semi Training-Free Cued Speech Recognition via MLLM-Driven Hand Modeling for Barrier-free Communication

Guanjie Huang, Danny Hin Kwok Tsang, Li Liu

专题命中 音频语音多模态 ：MLLM(title,abstract)；multimodal(abstract)；分类 eess.AS

Multimodal Large Language Models for Image, Text, and Speech Data Augmentation: A Survey

Ranjan Sapkota, Shaina Raza, Maged Shoman, Achyut Paudel, Manoj Karkee

专题命中 音频语音多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 52 pages

Adapting to the Unknown: Training-Free Audio-Visual Event Perception with Dynamic Thresholds

Eitan Shaar, Ariel Shaulov, Gal Chechik, Lior Wolf

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

MAG: Multi-Modal Aligned Autoregressive Co-Speech Gesture Generation without Vector Quantization

Binjie Liu, Lina Liu, Sanyi Zhang, Songen Gu, Yihao Zhi, Tianyi Zhu, Lei Yang, Long Ye

专题命中 音频语音多模态 ：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Survey of Adversarial Robustness in Multimodal Large Language Models

Chengze Jiang, Zhuangzhuang Wang, Minjing Dong, Jie Gui

专题命中 音频语音多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments 9 pages

Robust Audio-Visual Segmentation via Audio-Guided Visual Convergent Alignment

Chen Liu, Peike Li, Liying Yang, Dadong Wang, Lincheng Li, Xin Yu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by CVPR2025