arXivDaily arXiv每日学术速递 周一至周五更新

AI 大模型

多模态大模型

跨文本、图像、视频、音频等模态的大模型与学习方法。

2026-05-12 至 2026-05-12 共收录 13 信号源:cs.CV, cs.CL, cs.AI, cs.MM, eess.AS

1. 音频语音多模态 13 篇

2605.09906 2026-05-12 cs.AI cs.SD 88%

Separate First, Fuse Later: Mitigating Cross-Modal Interference in Audio-Visual LLMs Reasoning with Modality-Specific Chain-of-Thought

先分后融:通过模态特定的推理链减轻音频视觉大语言模型中的跨模态干扰

Xuanchen Li, Yuheng Lu, Chenrui Cui, Tianrui Wang, Zikang Huang, Yu Jiang, Long Zhou, Longbiao Wang, Jianwu Dang

机构 * Tianjin Key Laboratory of Cognitive Computing(天津认知计算实验室) Tianjin University(天津大学) Huiyan Technology Company, Ltd.(慧颜科技有限公司) Chinese Academy of Sciences(中国科学院) Tencent(腾讯)

专题命中 音频语音多模态 :cross-modal(title,abstract);audio-visual(title,abstract);分类 cs.AI

AI总结 本文提出SFFL框架,通过模态特定的推理链减少跨模态干扰,提升音频视觉问答的准确性和鲁棒性,实验显示在通用AVQA基准和跨模态幻觉基准上分别提升5.16%和11.17%。

详情

展开后加载摘要…

URL PDF HTML 收藏
2605.08723 2026-05-12 cs.CV cs.MM 86%

EAR: Enhancing Uni-Modal Representations for Weakly Supervised Audio-Visual Video Parsing

EAR:增强单模表示以实现弱监督音频视觉视频解析

Huilai Li, Xiaomeng Di, Ying Xing, Yonghao Dang, Yiming Wang, Jianqin Yin

机构 * School of Intelligent Engineering and Automation, Beijing University of Posts and Telecommunications(智能工程与自动化学院,北京邮电大学) State Grid Corporation of China(国家电网公司) School of Artificial Intelligence, Beijing University of Posts and Telecommunications(人工智能学院,北京邮电大学)

专题命中 音频语音多模态 :audio-visual(title,abstract);multi-modal(abstract);cross-modal(abstract);分类 cs.CV、cs.MM

AI总结 本文提出EAR框架,通过增强伪标签生成器和AVVP模型的单模表示,提升视频解析的时序定位性能。

详情

展开后加载摘要…

URL PDF HTML 收藏
2605.08762 2026-05-12 cs.SD cs.LG 85%

Omni-DeepSearch: A Benchmark for Audio-Driven Omni-Modal Deep Search

Omni-DeepSearch:一种基于音频的多模态深度搜索基准

Tao Yu, yiming ding, Shenghua Chai, Minghui Zhang, Zhongtian Luo, Xinming Wang, Xinlong Chen, Zhaolu Kang, Junhao Gong, Yuxuan Zhou, Haopeng Jin, Zhiqing Cui, Jiabing Yang, YiFan Zhang, Hongzhu Yi, Zheqi He, Xi Yang, Yan Huang, Liang Wang

机构 * CASIA UCAS BAAI Peking University(北京大学) Tsinghua University(清华大学)

专题命中 音频语音多模态 :omni-modal(title,abstract);multimodal(abstract);cross-modal(abstract)

AI总结 本文提出Omni-DeepSearch基准,用于评估基于音频的多模态深度搜索能力,通过多跳推理生成客观答案,结果显示该任务极具挑战性。

Comments 43 pages

详情

展开后加载摘要…

URL PDF HTML 收藏
2601.22143 2026-05-12 cs.GR cs.CV 83%

JUST-DUB-IT: Video Dubbing via Joint Audio-Visual Diffusion

JUST-DUB-IT: 通过联合音频-视觉扩散进行视频配音

Anthony Chen, Naomi Ken Korem, Gal Zeevi, Tavi Halperin, Matan Ben Yosef, Urska Jelercic, Ofir Bibi, Or Patashnik, Daniel Cohen-Or

机构 * Tel Aviv University(特拉维夫大学)

专题命中 音频语音多模态 :audio-visual(title,abstract);multi-modal(abstract);分类 cs.CV

AI总结 本文提出JUST-DUB-IT方法,利用轻量LoRA调整基础音频-视频扩散模型,实现视频到视频的高质量配音,提升视觉保真度和唇同步性能。

Comments Project webpage available at https://justdubit.github.io

详情

展开后加载摘要…

URL PDF HTML 收藏
2605.09468 2026-05-12 cs.MM 83%

Mitigating Multimodal Inconsistency via Cognitive Dual-Pathway Reasoning for Intent Recognition

通过认知双通路推理缓解多模态不一致:意图识别

Yifan Wang, Peiwu Wang, Yunxian Chi, Zhinan Gou, Kai Gao

专题命中 音频语音多模态 :multimodal(title,abstract);cross-modal(abstract);分类 cs.MM

AI总结 本文提出认知双通路推理框架,通过构建稳定语义基础和缓解高层语义冲突,提升多模态意图识别的准确性和鲁棒性。

Comments Accepted by ICMR 2026 (Main Track, Long Paper)

详情

展开后加载摘要…

URL PDF HTML 收藏
2605.08805 2026-05-12 cs.CV 83%

LightAVSeg: Lightweight Audio-Visual Segmentation

LightAVSeg: 轻量级音频视觉分割

Qing Zhong, Guodong Ding, Lingqiao Liu, Zaiwen Feng, Lin Yuanbo Wu, Angela Yao

机构 * College of Informatics, Huazhong Agricultural University, Wuhan, China(华中农业大学信息学院) School of Computing, National University of Singapore, Singapore(新加坡国立大学计算机学院) School of Computer Science, Adelaide University, Australia(阿德莱德大学计算机科学学院) School of Engineering, University of Warwick, Coventry, UK(沃里克大学工程学院) Zhejiang Yuexiu University, Shaoxing, China(浙江越秀大学)

专题命中 音频语音多模态 :audio-visual(title,abstract);cross-modal(abstract);分类 cs.CV

AI总结 本文提出LightAVSeg,通过解耦设计替代重注意力机制,实现线性交互成本,提升效率,实验显示其在MS3基准上达到50.4 mIoU,参数量仅为AVSegFormer的1/7。

Comments 15 pages, 8 figures, 6 tables, Accepted to ICML 2026

详情

展开后加载摘要…

URL PDF HTML 收藏
2604.24954 2026-05-12 cs.LG cs.AI cs.CV 81%

Nemotron 3 Nano Omni: Efficient and Open Multimodal Intelligence

Nemotron 3 Nano Omni:高效且开放的多模态智能

NVIDIA, :, Amala Sanjay Deshmukh, Kateryna Chumachenko, Tuomas Rintamaki, Matthieu Le, Tyler Poon, Danial Mohseni Taheri, Ilia Karmanov, Guilin Liu, Jarno Seppanen, Arushi Goel, Mike Ranzinger, Greg Heinrich, Guo Chen, Lukas Voegtle, Philipp Fischer, Timo Roman, Karan Sapra, Collin McCarthy, Shaokun Zhang, Fuxiao Liu, Hanrong Ye, Yi Dong, Mingjie Liu, Yifan Peng, Piotr Zelasko, Zhehuai Chen, Nithin Rao Koluguri, Nune Tadevosyan, Lilit Grigoryan, Ehsan Hosseini Asl, Pritam Biswas, Leili Tavabi, Yuanhang Su, Zhiding Yu, Peter Jin, Alexandre Milesi, Netanel Haber, Yao Xu, Sarah Amiraslani, Nabin Mulepati, Eric Tramel, Jaehun Jung, Ximing Lu, Brandon Cui, Jin Xu, Zhiqi Li, Shihao Wang, Yuanguo Kuang, Shaokun Zhang, Huck Yang, Boyi Li, Hongxu Yin, Song Han, Bilal Kartal, Pavlo Molchanov, Adi Renduchintala, Charles Wang, David Mosallanezhad, Soumye Singhal, Luis Vega, Katherine Cheung, Sreyan Ghosh, Yian Zhang, Alexander Bukharin, Venkat Srinivasan, Johnny Greco, Andre Manoel, Maarten Van Segbroeck, Suseella Panguliri, Rohit Watve, Divyanshu Kakwani, Shubham Pachori, Jeffrey Glick, Radha Sri-Tharan, Aileen Zaman, Khanh Nguyen, Shi Chen, Jiaheng Fang, Qing Miao, Wenfei Zhou, Yu Wang, Zaid Pervaiz Bhat, Varun Praveen, Arihant Jain, Ramanathan Arunachalam, Tomasz Kornuta, Ashton Sharabiani, Amy Shen, Wei Huang, Yi-Fu Wu, Ali Roshan Ghias, Huiying Li, Brian Yu, Nima Tajbakhsh, Chen Cui, Wenwen Gao, Li Ding, Terry Kong, Manoj Kilaru, Anahita Bhiwandiwalla, Marek Wawrzos, Daniel Korzekwa, Pablo Ribalta, Grzegorz Chlebus, Besmira Nushi, Ewa Dobrowolska, Maciej Jakub Mikulski, Kunal Dhawan, Steve Huang, Jagadeesh Balam, Yongqiang Wang, Nikolay Karpov, Valentin Mendelev, George Zelenfroynd, Meline Mkrtchyan, Qing Miao, Omri Almog, Bhavesh Pawar, Rameshwar Shivbhakta, Sudeep Sabnis, Ashrton Sharabiani, Negar Habibi, Geethapriya Venkataramani, Pamela Peng, Prerit Rodney, Serge Panev, Richard Mazzarese, Nicky Liu, Michael Fukuyama, Andrii Skliar, Roger Waleffe, Duncan Riach, Yunheng Zou, Jian Hu, Hao Zhang, Binfeng Xu, Yuhao Yang, Zuhair Ahmed, Alexandre Milesi, Carlo del Mundo, Chad Voegele, Zhiyu Cheng, Nave Assaf, Andrii Skliar, Daniel Afrimi, Natan Bagrov, Ran Zilberstein, Ofri Masad, Eugene Khvedchenia, Natan Bagrov, Borys Tymchenko, Tomer Asida, Daniel Afrimi, Parth Mannan, Victor Cui, Michael Evans, Katherine Luna, Jie Lou, Pinky Xu, Guyue Huang, Negar Habibi, Michael Boone, Pradeep Thalasta, Adeola Adesoba, Dina Yared, Christopher Parisien, Leon Derczynski, Shaona Ghosh, Wes Feely, Micah Schaffer, Radha Sri-Tharan, Jeffrey Glick, Barnaby Simkin, George Zelenfroynd, Tomasz Grzegorzek, Rishabh Garg, Aastha Jhunjhunwala, Sergei Kolchenko, Farzan Memarian, Haran Kumar, Shiv Kumar, Isabel Hulseman, Anjali Shah, Kari Briski, Padmavathy Subramanian, Joey Conway, Udi Karpas, Jane Polak Scowcroft, Annie Surla, Shilpa Ammireddy, Ellie Evans, Jesse Oliver, Tom Balough, Chia-Chih Chen, Sandip Bhaskar, Alejandra Rico, Bardiya Sadeghi, Seph Mard, Katherine Cheung, Meredith Price, Laya Sleiman, Saori Kaji, Wesley Helmholz, Wendy Quan, Michael Lightstone, Jonathan Cohen, Jian Zhang, Oleksii Kuchaiev, Boris Ginsburg, Jan Kautz, Eileen Long, Mohammad Shoeybi, Mostofa Patwary, Oluwatobi Olabiyi, Andrew Tao, Bryan Catanzaro, Udi Karpas

机构 * NVIDIA

专题命中 音频语音多模态 :multimodal(title,abstract);分类 cs.CV、cs.AI

AI总结 Nemotron 3 Nano Omni是首个原生支持音频输入的多模态模型,通过架构、数据和训练方法的改进,在所有模态上均实现了更准确的性能,同时提供更低的推理延迟和更高的吞吐量。

详情

展开后加载摘要…

URL PDF HTML 收藏
2605.08175 2026-05-12 cs.CV cs.AI 73%

KARMA-MV: A Benchmark for Causal Question Answering on Music Videos

KARMA-MV:音乐视频上的因果问答基准

Archishman Ghosh, Abhinaba Roy, Dorien Herremans

机构 * AMAAI Lab, Singapore University of Technology and Design(新加坡科技设计大学AMAAI实验室)

专题命中 音频语音多模态 :cross-modal(abstract);audio-visual(abstract);分类 cs.CV、cs.AI

AI总结 KARMA-MV是一个基于2682个YouTube音乐视频构建的大规模多选问答数据集,旨在测试模型整合时序音频视觉线索和视觉到音乐影响的能力,通过因果知识图谱方法提升音乐视频因果推理性能。

详情

展开后加载摘要…

URL PDF HTML 收藏
2605.09272 2026-05-12 cs.AI cs.CL cs.CV 67%

Towards Conversational Medical AI with Eyes, Ears and a Voice

面向有眼睛、耳朵和声音的对话式医疗AI

Meet Shah, Jason Gusdorf, Anil Palepu, Chunjong Park, Jack W. O'Sullivan, Vishnu Ravi, Tim Strother, Pavel Dubov, Aliya Rysbek, Toshiyuki Fukuzawa, Yana Lunts, Jan Freyberg, Michael B. Chang, Aniruddh Raghu, David Stutz, Devora Berlowitz, Eliseo Papa, Taylan Cemgil, JD Velasquez, Jack Chen, Arthur Chen, Doug Fritz, Charlie Taylor, Katya Tregubova, Jing Rong Lim, Richard Green, Sara Mahdavi, Mahvish Nagda, Jihyeon Lee, Craig Schiff, Liviu Panait, Sukhdeep Singh, Valentin Liévin, David G. T. Barrett, Hannah Gladman, Anna Cupani, Francesca Pietra, Uchechi Okereke, Katherine Tong, Clemens Meyer, Erwan Rolland, Mili Sanwalka, Michael D. Howell, Shixiang Shane Gu, Bibo Xu, Euan A. Ashley, S. M. Ali Eslami, Gregory Wayne, Pushmeet Kohli, Vivek Natarajan, Adam Rodman, Alan Karthikesalingam, Ryutaro Tanno

机构 * Google DeepMind(谷歌DeepMind) Google Research(谷歌研究) Beth Israel Deaconess Medical Center, Harvard Medical School(贝塞斯达医院, 哈佛医学院) Stanford University(斯坦福大学)

专题命中 音频语音多模态 :audio-visual(abstract);分类 cs.CV、cs.CL、cs.AI

AI总结 本文提出AI co-clinician系统,利用音频视频数据实现实时临床决策,通过TelePACES评估标准显示其在管理计划和诊断差异方面接近医生,但在体格检查和疾病特异性推理上仍有不足。

Comments Video examples are available on Youtube: https://youtu.be/y5Vaa_SN1t0, https://youtu.be/dC4icb75vLQ, and https://youtu.be/E7iEvWo-E6c

详情

展开后加载摘要…

URL PDF HTML 收藏
2605.09846 2026-05-12 cs.SD cs.AI 57%

ChladniSonify: A Visual-Acoustic Mapping Method for Chladni Patterns in New Media Art Creation

ChladniSonify:一种用于新媒介艺术创作中Chladni图案的视觉-听觉映射方法

Yakun Liu, Hai Luan, Dong Liu, Zhiyu Jin

机构 * Department of Composition(作曲系) Education Information Center(教育信息中心) Department of Musicology(音乐学系)

专题命中 音频语音多模态 :audio-visual(abstract);分类 cs.AI

AI总结 本文提出ChladniSonify,通过基于Kirchhoff-Love板理论的数值编程构建数据集并利用ANSYS仿真校准,采用轻量级CNN与CBAM实现高精度低延迟的Chladni图案分类,最终构建端到端系统将识别图案映射到对应正弦波频率,实现实时交互。

Comments 9 pages, 5 figures, IEEE conference format

详情

展开后加载摘要…

URL PDF HTML 收藏
2605.09656 2026-05-12 cs.RO 50%

ORICF -- Open Robotics Inference and Control Framework

ORICF -- 开源机器人推断与控制框架

Andrés Meseguer Valenzuela, Luís Miguel Bartolín Arnau

机构 * Instituto Tecnológico de Informática (ITI)(技术信息学院)

专题命中 音频语音多模态 :multimodal(abstract)

AI总结 本文提出ORICF框架,通过模块化设计实现多模态机器人推断流水线,支持边缘计算,降低计算开销和能耗,保持模块化和可重复性。

Comments Accepted in ICRA26 Workshop: 8th International Workshop on Robotics Software Engineering (RoSE 26)

详情

展开后加载摘要…

URL PDF HTML 收藏
2605.09434 2026-05-12 cs.DC cs.HC cs.LG 50%

PoHAR: Understanding Hyperlocal Human Activities with Pollution Sensor Networks

PoHAR:利用污染传感器网络理解超本地人类活动

Prasenjit Karmakar, Karthik Reddy, Sandip Chakraborty

机构 * Department of Computer Science and Engineering(计算机科学与工程系) Indian Institute of Technology Kharagpur(印度理工学院克拉格普尔)

专题命中 音频语音多模态 :audio-visual(abstract)

AI总结 本文提出PoHAR框架,通过冲突自由的数据复制原语、分层聚类和基于领导者的分组推断,实现室内超本地活动检测,实验显示在设备上检测室内活动和烹饪活动的准确率分别达到97.41%和99.68%。

Comments 8 pages, 8 figures, accepted to IEEE DCOSS-IoT 2026

详情

展开后加载摘要…

URL PDF HTML 收藏
2502.20213 2026-05-12 cs.LG cs.CY 50%

Mixture of Experts for Recognizing Depression from Interview and Reading Tasks

专家混合模型用于从访谈和阅读任务中识别抑郁症

Loukas Ilias, Dimitris Askounis

机构 * DSS Laboratory, School of ECE, National Technical University of Athens, Greece(国家技术大学雅典分校电子工程系DSS实验室)

专题命中 音频语音多模态 :multimodal(abstract)

AI总结 本文提出利用多模态融合和专家混合模型,从访谈和阅读任务的语音中识别抑郁症,实现87%的准确率和86.66%的F1分数。

Comments Accepted at ICASSP 2026

详情

展开后加载摘要…

URL PDF HTML 收藏