多模态大模型 - arXivDaily 专题

2003.07032 2020-10-26 eess.AS cs.SD eess.IV 83%

Multi-modal Multi-channel Target Speech Separation

Rongzhi Gu, Shi-Xiong Zhang, Yong Xu, Lianwu Chen, Yuexian Zou, Dong Yu

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments accepted in IEEE Journal of Selcted Topics in Signal Processing

2010.10839 2020-10-22 cs.CL 83%

TMT: A Transformer-based Modal Translator for Improving Multimodal Sequence Representations in Audio Visual Scene-aware Dialog

Wubo Li, Dongwei Jiang, Wei Zou, Xiangang Li

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CL

2009.02095 2020-10-02 eess.AS cs.LG cs.SD 83%

SEANet: A Multi-modal Speech Enhancement Network

Marco Tagliasacchi, Yunpeng Li, Karolis Misiunas, Dominik Roblek

专题命中音频语音多模态：multi-modal(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments Accepted to INTERSPEECH 2020

2009.09561 2020-09-22 cs.SD cs.LG eess.AS 83%

Correlating Subword Articulation with Lip Shapes for Embedding Aware Audio-Visual Speech Enhancement

Hang Chen, Jun Du, Yu Hu, Li-Rong Dai, Bao-Cai Yin, Chin-Hui Lee

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 eess.AS

Comments 34 pages, 8 figures

2006.07931 2020-08-10 eess.AS cs.DB cs.SD 83%

Solos: A Dataset for Audio-Visual Music Analysis

Juan F. Montesinos, Olga Slizovskaia, Gloria Haro

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 eess.AS

Comments Rephrased some sentenced. Explanation about OpenPose. Minor grammatical errors

2007.04687 2020-07-14 cs.CV 83%

Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision

Peng Wu, Jing Liu, Yujia Shi, Yujia Sun, Fangtao Shao, Zhaoyang Wu, Zhiwei Yang

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments To appear in ECCV 2020

1912.00581 2020-06-17 cs.AI cs.HC q-bio.NC 83%

Optimality and limitations of audio-visual integration for cognitive systems

W. Paul Boyce, Tony Lindsay, Arkady Zgonnikov, Ignacio Rano, KongFatt Wong-Lin

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI

Comments 20 pages, 6 figures, 1 table 16/06/2020: Updated version includes expanded discussion and addition of new references. Also updated author affiliation information. This version has been accepted for publication with Frontiers

2005.09297 2020-05-20 eess.AS cs.LG 83%

Should we hard-code the recurrence concept or learn it instead ? Exploring the Transformer architecture for Audio-Visual Speech Recognition

George Sterpu, Christian Saam, Naomi Harte

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 eess.AS

Comments Submitted to INTERSPEECH 2020

2001.11406 2020-02-06 cs.MM cs.LG eess.IV 83%

NAViDAd: A No-Reference Audio-Visual Quality Metric Based on a Deep Autoencoder

Helard Martinez, M. C. Farias, A. Hines

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM

Comments 5 pages

Journal ref 2019 27th European Signal Processing Conference (EUSIPCO), IEEE, 2019, pp 1-5

2002.00163 2020-02-04 cs.CL 83%

Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

Zekang Li, Zongjia Li, Jinchao Zhang, Yang Feng, Cheng Niu, Jie Zhou

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CL

Comments Accepted by AAAI2020 DSTC8 workshop. Ranked 1st in DSTC8-AVSD track

1909.06957 2019-09-18 cs.CV 83%

Multimodal Deep Models for Predicting Affective Responses Evoked by Movies

Ha Thi Phuong Thao, Dorien Herremans, Gemma Roig

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments 10 pages, 7 figures, Preprint accepted for publication in the Proceedings of the 2nd International Workshop on Computer Vision for Physiological Measurement as part of ICCV. Seoul, South Korea. 2019

Journal ref Proceedings of the 2nd International Workshop on Computer Vision for Physiological Measurement as part of ICCV. Seoul, South Korea. 2019

1906.10096 2019-06-25 cs.CV 83%

Audio-Visual Kinship Verification

Xiaoting Wu, Eric Granger, Xiaoyi Feng

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments 12 pages

1905.13694 2019-06-03 cs.CV 83%

Multimodal Joint Emotion and Game Context Recognition in League of Legends Livestreams

Charles Ringer, James Alfred Walker, Mihalis A. Nicolaou

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

Comments 8 Pages, IEEE Conference on Games 2019

1809.01728 2019-05-02 eess.AS cs.LG cs.SD eess.IV stat.ML 83%

Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition

George Sterpu, Christian Saam, Naomi Harte

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments In ICMI'18, October 16-20, 2018, Boulder, CO, USA. Equation (2) corrected on this version

1904.05876 2019-04-12 cs.CV cs.AI cs.CL cs.LG cs.SD eess.AS 83%

A Simple Baseline for Audio-Visual Scene-Aware Dialog

Idan Schwartz, Alexander Schwing, Tamir Hazan

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to CVPR 2019

1812.02872 2018-12-10 cs.CV 83%

An Attempt towards Interpretable Audio-Visual Video Captioning

Yapeng Tian, Chenxiao Guan, Justin Goodman, Marc Moore, Chenliang Xu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 11 pages, 4 figures

1811.05760 2018-11-15 eess.AS cs.CL cs.CV cs.MM cs.SD 83%

A Multimodal Approach towards Emotion Recognition of Music using Audio and Lyrical Content

Aniruddha Bhattacharya, K. V. Kadambari

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments 6 pages

1612.08727 2018-08-09 cs.MM cs.SD 83%

Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications

Bochen Li, Xinzhao Liu, Karthik Dinesh, Zhiyao Duan, Gaurav Sharma

专题命中音频语音多模态：multimodal(title)；multi-modal(abstract)；audio-visual(abstract)；分类 cs.MM

Comments 14 pages, 10 figures

1803.08842 2018-03-26 cs.CV 83%

Audio-Visual Event Localization in Unconstrained Videos

Yapeng Tian, Jing Shi, Bochen Li, Zhiyao Duan, Chenliang Xu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 23 pages, 7 figures

1803.04840 2018-03-14 cs.CV 83%

Resource aware design of a deep convolutional-recurrent neural network for speech recognition through audio-visual sensor fusion

Matthijs Van keirsbilck, Bert Moons, Marian Verhelst

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Tech. report

1712.09680 2018-03-02 cs.SD eess.AS 83%

A Light-Weight Multimodal Framework for Improved Environmental Audio Tagging

Juncheng Li, Yun Wang, Joseph Szurley, Florian Metze, Samarjit Das

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments 5 pages, 3 figures, Accepted and to appear at ICASSP 2018

1711.08102 2017-12-12 cs.CV 83%

CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation

Wangli Hao, Zhaoxiang Zhang, He Guan

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments Have some problems need to be handled

1711.11017 2017-11-30 cs.AI cs.CL cs.CV cs.RO cs.SD eess.AS 83%

HoME: a Household Multimodal Environment

Simon Brodeur, Ethan Perez, Ankesh Anand, Florian Golemo, Luca Celotti, Florian Strub, Jean Rouat, Hugo Larochelle, Aaron Courville

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Presented at NIPS 2017's Visually-Grounded Interaction and Language Workshop

1706.05739 2017-11-01 cs.CV 83%

3D Convolutional Neural Networks for Cross Audio-Visual Matching Recognition

Amirsina Torfi, Seyed Mehdi Iranmanesh, Nasser M. Nasrabadi, Jeremy Dawson

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Journal ref IEEE Access (Year: 2017, Volume: PP, Issue: 99 )

1704.06913 2017-10-19 cs.CL cs.LG 83%

Learning weakly supervised multimodal phoneme embeddings

Rahma Chaabouni, Ewan Dunbar, Neil Zeghidour, Emmanuel Dupoux

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

2312.05265 2023-12-12 cs.AI cs.CR cs.CV cs.LG eess.AS 83%

Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features

Anderson Augusma, Dominique Vaufreydaz, Frédérique Letué

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Journal ref ICMI '23: INTERNATIONAL CONFERENCE ON MULTIMODAL INTERACTION, Oct 2023, Paris, France. pp.750-754

2310.05938 2023-10-11 cs.CV cs.AI cs.SD eess.AS 83%

Component attention network for multimodal dance improvisation recognition

Jia Fu, Jiarui Tan, Wenjie Yin, Sepideh Pashami, Mårten Björkman

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted to 25th ACM International Conference on Multimodal Interaction (ICMI 2023)

2201.11664 2022-01-28 cs.CV cs.AI cs.LG cs.MM 83%

Team Yao at Factify 2022: Utilizing Pre-trained Models and Co-attention Networks for Multi-Modal Fact Verification

Wei-Yao Wang, Wen-Chih Peng

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM；multimodal(comments)

Comments Accepted by AAAI 2022 De-Factify Workshop: First Workshop on Multimodal Fact-Checking and Hate Speech Detection

2607.24821 2026-07-29 cs.MM cs.CV cs.SD 新提交 82%

AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities

AVE-Compass：迈向音频-视频编辑能力的整体评估

Yuqing Wen, Yukai Huang, Qianqian Xie, Jiangtao Wu, Yibin Lin, Yikai Gu, Jialu Chen, Yuanxing Zhang, Jiaheng Liu

专题命中音频语音多模态：MLLM(abstract,abstract_cn)；cross-modal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM

AI总结研究针对现有视频编辑基准未充分考虑视听耦合问题，提出AVE-Compass基准及AVE-Agent框架，通过多种评估方式评估编辑能力，能分解复杂指令迭代改进结果，提升了跨模态编辑等方面表现及感知质量。

2510.06371 2026-05-13 cs.CL cs.AI 82%

OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QA

OASIS：一个多语言多模态数据集用于文化导向的语音视觉问答

Firoj Alam, Ali Ezzat Shahroor, Md. Arid Hasan, Zien Sheikh Ali, Hunzalah Hassan Bhatti, Mohamed Bayan Kmainasi, Shammur Absar Chowdhury, Basel Mousi, Fahim Dalvi, Nadir Durrani, Natasa Milic-Frayling

机构 * Qatar Computing Research Institute（卡塔尔计算研究 institute）

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI；multimodal foundation model(comments)

AI总结 OASIS是一个多语言多模态数据集，涵盖图像、文本和语音，旨在评估模型在现实场景中的文化导向推理能力，包含0.92M真实图像和14.8M问答对。

Comments Multimodal Foundation Models, Large Language Models, Native, Multilingual, Language Diversity, Contextual Understanding, Culturally Informed

1. 音频语音多模态 4549 篇

Multi-modal Multi-channel Target Speech Separation

TMT: A Transformer-based Modal Translator for Improving Multimodal Sequence Representations in Audio Visual Scene-aware Dialog

SEANet: A Multi-modal Speech Enhancement Network

Correlating Subword Articulation with Lip Shapes for Embedding Aware Audio-Visual Speech Enhancement

Solos: A Dataset for Audio-Visual Music Analysis

Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision

Optimality and limitations of audio-visual integration for cognitive systems

Should we hard-code the recurrence concept or learn it instead ? Exploring the Transformer architecture for Audio-Visual Speech Recognition

NAViDAd: A No-Reference Audio-Visual Quality Metric Based on a Deep Autoencoder

Bridging Text and Video: A Universal Multimodal Transformer for Video-Audio Scene-Aware Dialog

Multimodal Deep Models for Predicting Affective Responses Evoked by Movies

Audio-Visual Kinship Verification

Multimodal Joint Emotion and Game Context Recognition in League of Legends Livestreams

Attention-based Audio-Visual Fusion for Robust Automatic Speech Recognition

A Simple Baseline for Audio-Visual Scene-Aware Dialog

An Attempt towards Interpretable Audio-Visual Video Captioning

A Multimodal Approach towards Emotion Recognition of Music using Audio and Lyrical Content

Creating A Multi-track Classical Musical Performance Dataset for Multimodal Music Analysis: Challenges, Insights, and Applications

Audio-Visual Event Localization in Unconstrained Videos

Resource aware design of a deep convolutional-recurrent neural network for speech recognition through audio-visual sensor fusion

A Light-Weight Multimodal Framework for Improved Environmental Audio Tagging

CMCGAN: A Uniform Framework for Cross-Modal Visual-Audio Mutual Generation

HoME: a Household Multimodal Environment

3D Convolutional Neural Networks for Cross Audio-Visual Matching Recognition

Learning weakly supervised multimodal phoneme embeddings

Multimodal Group Emotion Recognition In-the-wild Using Privacy-Compliant Features

Component attention network for multimodal dance improvisation recognition

Team Yao at Factify 2022: Utilizing Pre-trained Models and Co-attention Networks for Multi-Modal Fact Verification

AVE-Compass: Towards Holistic Evaluation for Audio-Video Editing Abilities

OASIS: A Multilingual and Multimodal Dataset for Culturally Grounded Spoken Visual QA