多模态大模型

Multimodal Relational Tensor Network for Sentiment and Emotion Classification

Saurav Sahay, Shachi H Kumar, Rui Xia, Jonathan Huang, Lama Nachman

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Seeing Voices and Hearing Faces: Cross-modal biometric matching

Arsha Nagrani, Samuel Albanie, Andrew Zisserman

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 cs.CV

Comments To appear in: IEEE Computer Vision and Pattern Recognition (CVPR), 2018

Deep Multimodal Learning for Emotion Recognition in Spoken Language

Yue Gu, Shuhong Chen, Ivan Marsic

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments ICASSP 2018

End2You -- The Imperial Toolkit for Multimodal Profiling by End-to-End Learning

Panagiotis Tzirakis, Stefanos Zafeiriou, Bjorn W. Schuller

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV

Multimodal Attribute Extraction

Robert L. Logan, Samuel Humeau, Sameer Singh

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments AKBC 2017 Workshop Paper

Deep Cross-Modal Correlation Learning for Audio and Lyrics in Music Retrieval

Yi Yu, Suhua Tang, Francisco Raposo, Lei Chen

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 eess.AS

Multimodal Signal Processing and Learning Aspects of Human-Robot Interaction for an Assistive Bathing Robot

A. Zlatintsi, I. Rodomagoulakis, P. Koutras, A. C. Dometios, V. Pitsikalis, C. S. Tzafestas, P. Maragos

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM

Sympathy Begins with a Smile, Intelligence Begins with a Word: Use of Multimodal Features in Spoken Human-Robot Interaction

Jekaterina Novikova, Christian Dondrup, Ioannis Papaioannou, Oliver Lemon

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments Robo-NLP workshop at ACL 2017. 9 pages, 5 figures, 6 tables

EM Algorithms for Weighted-Data Clustering with Application to Audio-Visual Scene Analysis

Israel D. Gebru, Xavier Alameda-Pineda, Florence Forbes, Radu Horaud

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Comments 14 pages, 4 figures, 4 tables

Journal ref IEEE Transactions on Pattern Analysis and Machine Intelligence, volume 38, number 12, 2402 - 2415, 2016

Training an Interactive Humanoid Robot Using Multimodal Deep Reinforcement Learning

Heriberto Cuayáhuitl, Guillaume Couly, Clément Olalainty

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Comments NIPS Workshop on Future of Interactive Learning Machines, 2016

Kernel-based Sensor Fusion with Application to Audio-Visual Voice Activity Detection

David Dov, Ronen Talmon, Israel Cohen

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Lattice-Based Graded Logic: a Multimodal Approach

Philippe Chatalic, Christine Froidevaux

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Comments Appears in Proceedings of the Eighth Conference on Uncertainty in Artificial Intelligence (UAI1992)

A Multimodal Biometric System Using Linear Discriminant Analysis For Improved Performance

Aamir Khan, Muhammad Farhan, Aasim Khurshid, Adeel Akram

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV

Journal ref IJCSI International Journal of Computer Science Issues, Vol. 8, Issue 6, No 2, 2011, 122-127

Cognitive Principles in Robust Multimodal Interpretation

J. Y. Chai, Z. Prasov, S. Qu

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Journal ref Journal Of Artificial Intelligence Research, Volume 27, pages 55-83, 2006

A Study on Potential of Integrating Multimodal Interaction into Musical Conducting Education

Gilbert Phuah Leong Siang, Nor Azman Ismail, Pang Yee Yong

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.MM

Comments http://www.journalofcomputing.org

Journal ref Journal of Computing, Volume 2, Issue 5, May 2010

Speech Dialogue with Facial Displays: Multimodal Human-Computer Conversation

Katashi Nagao, Akikazu Takeuchi

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments 8 pages, Postscript file, UNIX compressed, uuencoded

TabPFN beyond Tabular Data: Calibration and Accuracy on Multimodal Embeddings

超越表格数据的TabPFN：多模态嵌入的校准与准确性

Jingxiang Zhang, Lujia Zhong, Zijie Zhu, Shuo Huang, Yuang Xu

专题命中 音频语音多模态 ：multimodal(title,abstract)

AI总结 研究少样本多模态分类中轻量级头部校准不佳问题，核心方法是评估TabPFN作为即插即用分类头。贡献在于其在多数据集、多编码器和多模态评估中表现优异，能提升校准并保持准确率，为校准敏感多模态分类提供指导。

Comments 19 pages, 13 figures, 10 tables. Jingxiang Zhang and Lujia Zhong contributed equally. Code: https://github.com/Jingxiang-Zhang/tabpfn-multimodal-embeddings

FOCAL: A Novel Benchmarking Technique for Multi-modal Agents

FOCAL：多模态代理的新型基准测试技术

Anupam Purwar, Aditya Choudhary

专题命中 音频语音多模态 ：multi-modal(title,abstract)

AI总结 FOCAL提出了一种新型基准测试技术，用于评估多模态代理的端到端推理能力、误差传播及对话效果。

Comments We present a framework for evaluation of Multi-modal Agents consisting of Voice-to-voice model components viz. Text to Speech (TTS), Retrieval Augmented Generation (RAG) and Speech-to-text (STT)

Wan-Streamer v0.1: End-to-end Real-time Interactive Foundation Models

Wan-Streamer v0.1: 端到端实时交互基础模型

Lianghua Huang, Zhi-Fan Wu, Wei Wang, Yupeng Shi, Mengyang Feng, Junjie He, Chen-Wei Xie, Yu Liu, Jingren Zhou, Ang Wang, Bang Zhang, Baole Ai, Chen Liang, Cheng Yu, Chongyang Zhong, Jinwei Qi, Kai Zhu, Pandeng Li, Peng Zhang, Wenyuan Zhang, Xinhua Cheng, Yitong Huang, Yun Zheng, Yuzheng Wang, Zoubin Bi

机构 * Alibaba Group（阿里巴巴集团） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

AI总结 提出Wan-Streamer，一种原生流式、端到端的交互基础模型，通过单一Transformer联合建模语言、音频和视频，实现低延迟全双工音视频交互，无需外部模块。

Comments Website: https://wan-streamer.com

JAM-Flow: Joint Audio-Motion Synthesis with Flow Matching

JAM-Flow：联合音频-运动合成与流匹配

Mingi Kwon, Joonghyuk Shin, Jaeseok Jung, Jaesik Park, Youngjung Uh

机构 * Yonsei University（延世大学） ； CineLingo ； Seoul National University（首尔国立大学） 纠错

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、eess.AS

AI总结 本文提出JAM-Flow框架，通过流匹配和多模态扩散变压器架构，实现面部运动与语音的联合合成与条件生成，支持文本到说话人生成、音频驱动动画等多种任务。

Comments project page: https://joonghyuk.com/jamflow-web Under review. Preprint published on arXiv

Keep What Audio Cannot Say: Context-Preserving Token Pruning for Omni-LLMs

保留音频无法表达的内容：面向多模态大语言模型的上下文保留令牌修剪

Chaeyoung Jung, Kyeongha Rho, Joon Son Chung

机构 * Korea Advanced Institute of Science and Technology（韩国科学技术院） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

AI总结 本文提出ContextGuard框架，通过保留音频-视觉上下文并去除跨模态冗余，实现多模态大语言模型的高效令牌修剪，实验表明其在多个基准测试中表现优异，能有效减少输入令牌数量。

Stage-Adaptive Reliability Modeling for Continuous Valence-Arousal Estimation

面向连续情感估值的阶段自适应可靠性建模

Yubeen Lee, Sangeun Lee, Junyeop Cha, Eunil Park

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.AI、cs.MM

AI总结 本文提出SAGE框架，通过阶段自适应可靠性建模提升多模态环境下连续情感估值的稳定性与准确性。

Comments 8 pages, 3 figures, 2 pages

Integrating Spatial and Semantic Embeddings for Stereo Sound Event Localization in Videos

Davide Berghi, Philip J. B. Jackson

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.AI、eess.AS

Comments arXiv admin note: substantial text overlap with arXiv:2507.04845

Is Extending Modality The Right Path Towards Omni-Modality?

Tinghui Zhu, Kai Zhang, Muhao Chen, Yu Su

机构 * University of California, Davis（加州大学戴维斯分校） ； The Ohio State University（俄亥俄州立大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；multi-modal(abstract)；omni-modal(abstract)；分类 cs.CV、cs.CL

Bridging The Multi-Modality Gaps of Audio, Visual and Linguistic for Speech Enhancement

Meng-Ping Lin, Jen-Cheng Hou, Chia-Wei Chen, Shao-Yi Chien, Jun-Cheng Chen, Xugang Lu, Yu Tsao

机构 * National Taiwan University（国立台湾大学） ； Academia Sinica（中国学术.sinica） ； National Institute of Information and Communications Technology（信息与通信技术国家研究所） 纠错

专题命中 音频语音多模态 ：multi-modal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.MM、eess.AS

Multiscale Adaptive Conflict-Balancing Model For Multimedia Deepfake Detection

Zihan Xiong, Xiaohua Wu, Lei Chen, Fangqi Lou

机构 * University of Electronic Science and Technology of China（电子科学与技术大学） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.AI

Comments 9 pages,ICMR accepted

Hearing and Seeing Through CLIP: A Framework for Self-Supervised Sound Source Localization

Sooyoung Park, Arda Senocak, Joon Son Chung

机构 * School of Electrical Engineering, KAIST, South Korea（韩国延世大学电气工程学院） ； ETRI, South Korea（韩国电子技术研究院） 纠错

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；multimodal foundation model(abstract)；分类 cs.CV、eess.AS

Comments Journal Extension of WACV 2024 paper (arXiv:2311.04066). Code is available at ACL-SSL" target="_blank" rel="noopener">https://github.com/swimmiing/ACL-SSL

Lyra: An Efficient and Speech-Centric Framework for Omni-Cognition

Zhisheng Zhong, Chengyao Wang, Yuqi Liu, Senqiao Yang, Longxiang Tang, Yuechen Zhang, Jingyao Li, Tianyuan Qu, Yanwei Li, Yukang Chen, Shaozuo Yu, Sitong Wu, Eric Lo, Shu Liu, Jiaya Jia

专题命中 音频语音多模态 ：multimodal(abstract)；multi-modal(abstract)；MLLM(abstract)；分类 cs.CV、cs.MM

Comments Tech report

Tackling Data Bias in MUSIC-AVQA: Crafting a Balanced Dataset for Unbiased Question-Answering

Xiulong Liu, Zhikang Dong, Peng Zhang

专题命中 音频语音多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL、cs.AI

PandaGPT: One Model To Instruction-Follow Them All

Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, Deng Cai

专题命中 音频语音多模态 ：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Technical report, work in progress. Our project page is at https://panda-gpt.github.io/