多模态大模型 - arXivDaily 专题

2202.07360 2022-02-16 cs.HC cs.CV 79%

Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle

Abdul Rafey Aftab, Michael von der Beeck

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV

Journal ref 27th International Conference on Intelligent User Interfaces (IUI '22), March 22--25, 2022, Helsinki, Finland

2202.06507 2022-02-15 eess.AS cs.LG cs.SD q-bio.QM 79%

EMGSE: Acoustic/EMG Fusion for Multimodal Speech Enhancement

Kuan-Chen Wang, Kai-Chun Liu, Hsin-Min Wang, Yu Tsao

专题命中音频语音多模态：multimodal(title,abstract)；分类 eess.AS

Comments 5 pages, 4 figures, and 3 tables

2202.06218 2022-02-15 cs.LG cs.CL 79%

Emotion Based Hate Speech Detection using Multimodal Learning

Aneri Rana, Sonali Jha

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL

2202.05961 2022-02-15 cs.CV eess.IV 79%

Audio-Visual Fusion Layers for Event Type Aware Video Recognition

Arda Senocak, Junsik Kim, Tae-Hyun Oh, Hyeonggon Ryu, Dingzeyu Li, In So Kweon

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

2112.00158 2022-01-28 eess.AS 79%

Representation learning through cross-modal conditional teacher-student training for speech emotion recognition

Sundararajan Srinivasan, Zhaocheng Huang, Katrin Kirchhoff

专题命中音频语音多模态：cross-modal(title)；multimodal(abstract)；分类 eess.AS

Comments Accepted for publication at IEEE ICASSP 2022

2112.14088 2021-12-30 cs.CV 79%

Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation

Philipp Harzig, Moritz Einfalt, Rainer Lienhart

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

2007.04954 2021-12-30 cs.CV cs.GR cs.LG cs.RO 79%

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

Chuang Gan, Jeremy Schwartz, Seth Alter, Damian Mrowca, Martin Schrimpf, James Traer, Julian De Freitas, Jonas Kubilius, Abhishek Bhandwaldar, Nick Haber, Megumi Sano, Kuno Kim, Elias Wang, Michael Lingelbach, Aidan Curtis, Kevin Feigelis, Daniel M. Bear, Dan Gutfreund, David Cox, Antonio Torralba, James J. DiCarlo, Joshua B. Tenenbaum, Josh H. McDermott, Daniel L. K. Yamins

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV

Comments Oral Presentation at NeurIPS 21 Datasets and Benchmarks Track. Project page: http://www.threedworld.org

2112.12409 2021-12-24 cs.CV 79%

InstaIndoor and Multi-modal Deep Learning for Indoor Scene Recognition

Andreea Glavan, Estefania Talavera

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV

2112.09172 2021-12-20 cs.CV cs.LG eess.IV 79%

An Audio-Visual Dataset and Deep Learning Frameworks for Crowded Scene Classification

Lam Pham, Dat Ngo, Phu X. Nguyen, Truong Hoang, Alexander Schindler

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

2112.02749 2021-12-07 cs.CV 79%

One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning

Suzhen Wang, Lincheng Li, Yu Ding, Xin Yu

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

Comments Accepted by AAAI 2022

Journal ref AAAI 2022

2106.15909 2021-11-30 eess.AS cs.SD 79%

Effect of acoustic scene complexity and visual scene representation on auditory perception in virtual audio-visual environments

Stefan Fichna, Thomas Biberger, Bernhard U. Seeber, Stephan D. Ewert

专题命中音频语音多模态：audio-visual(title,abstract)；分类 eess.AS

Comments Accepted publication in Proceedings of 3DA 2021 International Conference on Immersive and 3D Audio

2111.08702 2021-11-18 cs.CV cs.CR 79%

The Multiscenario Multienvironment BioSecure Multimodal Database (BMDB)

Javier Ortega-Garcia, Julian Fierrez, Fernando Alonso-Fernandez, Javier Galbally, Manuel R Freire, Joaquin Gonzalez-Rodriguez, Carmen Garcia-Mateo, Jose-Luis Alba-Castro, Elisardo Gonzalez-Agulla, Enrique Otero-Muras, Sonia Garcia-Salicetti, Lorene Allano, Bao Ly-Van, Bernadette Dorizzi, Josef Kittler, Thirimachos Bourlai, Norman Poh, Farzin Deravi, Ming NR Ng, Michael Fairhurst, Jean Hennebert, Andreas Humm, Massimo Tistarelli, Linda Brodo, Jonas Richiardi, Andrezj Drygajlo, Harald Ganster, Federico M Sukno, Sri-Kaushik Pavani, Alejandro Frangi, Lale Akarun, Arman Savran

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV

Comments Published at IEEE Transactions on Pattern Analysis and Machine Intelligence journal

2111.03472 2021-11-08 cs.CR cs.CV eess.IV 79%

BiosecurID: a multimodal biometric database

Julian Fierrez, Javier Galbally, Javier Ortega-Garcia, Manuel R Freire, Fernando Alonso-Fernandez, Daniel Ramos, Doroteo Torre Toledano, Joaquin Gonzalez-Rodriguez, Juan A Siguenza, Javier Garrido-Salas, E Anguiano, Guillermo Gonzalez-de-Rivera, Ricardo Ribalda, Marcos Faundez-Zanuy, JA Ortega, Valentín Cardeñoso-Payo, A Viloria, Carlos E Vivaracho, Q Isaac Moro, Juan J Igarza, J Sanchez, Inmaculada Hernaez, Carlos Orrite-Urunuela, Francisco Martinez-Contreras, Juan José Gracia-Roche

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV

Comments Published at Pattern Analysis and Applications journal

2104.13276 2021-11-01 cs.SD cs.DB cs.IR cs.LG eess.AS 79%

MULTIMODAL ANALYSIS: Informed content estimation and audio source separation

Gabriel Meseguer-Brocal

专题命中音频语音多模态：multimodal(title,abstract)；分类 eess.AS

Comments Ph.D. dissertation. Thesis supervisor: Geoffroy Peeters. Jury:Laurent Girin, Gaël Richard, Rachel Bittner, Elena Cabrio, Bruno Gas, Perfecto Herrera Boyer, Antoine Liutkus

2110.13570 2021-10-28 cs.CV 79%

Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition

Siyang Song, Zilong Shao, Shashank Jaiswal, Linlin Shen, Michel Valstar, Hatice Gunes

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

Comments Submitted to IJCV

2110.02500 2021-10-07 eess.AS 79%

MediumVC: Any-to-any voice conversion using synthetic specific-speaker speeches as intermedium features

Yewei Gu, Zhenyu Zhang, Xiaowei Yi, Xianfeng Zhao

专题命中音频语音多模态：any-to-any(title,abstract)；分类 eess.AS

Comments 5 pages, 2 figures

2109.09536 2021-09-21 cs.CV cs.LG 79%

Audio-Visual Speech Recognition is Worth 32$\times$32$\times$8 Voxels

Dmitriy Serdyuk, Otavio Braga, Olivier Siohan

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

Comments 7 pages, 2 figures, 4 tables. A draft for a paper accepted to ASRU workshop

2109.00181 2021-09-02 cs.SD cs.AI 79%

CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations

Hang Li, Yu Kang, Tianqiao Liu, Wenbiao Ding, Zitao Liu

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.AI

Comments The 2021 Conference on Empirical Methods in Natural Language Processing

2008.07191 2021-09-01 eess.AS cs.LG cs.SD 79%

Deep Variational Generative Models for Audio-visual Speech Separation

Viet-Nhat Nguyen, Mostafa Sadeghi, Elisa Ricci, Xavier Alameda-Pineda

专题命中音频语音多模态：audio-visual(title,abstract)；分类 eess.AS

Comments Accepted to the 31st IEEE International Workshop on Machine Learning for Signal Processing (MLSP), Oct. 25-28, 2021, Gold Coast, Queensland, Australia

2011.07755 2021-08-31 eess.AS 79%

Audio-visual Multi-channel Integration and Recognition of Overlapped Speech

Jianwei Yu, Shi-Xiong Zhang, Bo Wu, Shansong Liu, Shoukang Hu, Mengzhe Geng, Xunying Liu, Helen Meng, Dong Yu

专题命中音频语音多模态：audio-visual(title,abstract)；分类 eess.AS

Comments TASLP 2021

2101.10803 2021-08-18 cs.CV 79%

ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning

Sangho Lee, Jiwan Chung, Youngjae Yu, Gunhee Kim, Thomas Breuel, Gal Chechik, Yale Song

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

Comments Published to ICCV2021

2108.04343 2021-08-11 cs.AI cs.LG 79%

Towards a Generic Multimodal Architecture for Batch and Streaming Big Data Integration

Siham Yousfi, Maryem Rhanoui, Dalila Chiadmi

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.AI

Journal ref Journal of Computer Science, Volume 15 No. 1, 2019, 207-220

2012.06170 2021-08-10 cs.CV 79%

ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction

Samyak Jain, Pradeep Yarlagadda, Shreyank Jyoti, Shyamgopal Karthik, Ramanathan Subramanian, Vineet Gandhi

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV

Comments Appearing in the proceedings of the 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS 2021) (camera-ready version)

2104.10557 2021-08-04 cs.IR cs.MM 79%

Deep Music Retrieval for Fine-Grained Videos by Exploiting Cross-Modal-Encoded Voice-Overs

Tingtian Li, Zixun Sun, Haoruo Zhang, Jin Li, Ziming Wu, Hui Zhan, Yipeng Yu, Hengcan Shi

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.MM

Comments accepted by ACM SIGIR '21

2012.00096 2021-08-03 cs.LG cs.CL 79%

Multi-Modal Detection of Alzheimer's Disease from Speech and Text

Amish Mittal, Sourav Sahoo, Arnhav Datar, Juned Kadiwala, Hrithwik Shalu, Jimson Mathew

专题命中音频语音多模态：multi-modal(title)；multimodal(abstract)；分类 cs.CL

Comments 9 pages, 3 figures, Accepted in BIOKDD 2021

2107.06592 2021-07-27 eess.AS cs.SD eess.IV 79%

Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection

Ruijie Tao, Zexu Pan, Rohan Kumar Das, Xinyuan Qian, Mike Zheng Shou, Haizhou Li

专题命中音频语音多模态：audio-visual(title,abstract)；分类 eess.AS

Comments ACM Multimedia 2021

2007.08855 2021-07-21 cs.NE cs.AI q-bio.NC 79%

A Biologically Plausible Audio-Visual Integration Model for Continual Learning

Wenjie Chen, Fengtong Du, Ye Wang, Lihong Cao

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.AI

Comments Accepted by 2021 International Joint Conference on Neural Networks

1702.07452 2021-07-15 cs.MM cs.NI 79%

Software Defined Media: Virtualization of Audio-Visual Services

Manabu Tsukada, Keiko Ogawa, Masahiro Ikeda, Takuro Sone, Kenta Niwa, Shoichiro Saito, Takashi Kasuya, Hideki Sunahara, Hiroshi Esaki

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.MM

Comments IEEE International Conference on Communications (ICC2017), Paris, France, 21-25 May 2017

2107.01571 2021-07-06 cs.CL 79%

Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model

Zhiqi Huang, Fenglin Liu, Xian Wu, Shen Ge, Helin Wang, Wei Fan, Yuexian Zou

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL

Comments AAAI 2021

2107.01569 2021-07-06 cs.CL cs.LG 79%

Cross-Modal Transformer-Based Neural Correction Models for Automatic Speech Recognition

Tomohiro Tanaka, Ryo Masumura, Mana Ihori, Akihiko Takashima, Takafumi Moriya, Takanori Ashihara, Shota Orihashi, Naoki Makishima

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CL

Comments Accepted to Interspeech 2021

1. 音频语音多模态 4559 篇

Multimodal Driver Referencing: A Comparison of Pointing to Objects Inside and Outside the Vehicle

EMGSE: Acoustic/EMG Fusion for Multimodal Speech Enhancement

Emotion Based Hate Speech Detection using Multimodal Learning

Audio-Visual Fusion Layers for Event Type Aware Video Recognition

Representation learning through cross-modal conditional teacher-student training for speech emotion recognition

Synchronized Audio-Visual Frames with Fractional Positional Encoding for Transformers in Video-to-Text Translation

ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulation

InstaIndoor and Multi-modal Deep Learning for Indoor Scene Recognition

An Audio-Visual Dataset and Deep Learning Frameworks for Crowded Scene Classification

One-shot Talking Face Generation from Single-speaker Audio-Visual Correlation Learning

Effect of acoustic scene complexity and visual scene representation on auditory perception in virtual audio-visual environments

The Multiscenario Multienvironment BioSecure Multimodal Database (BMDB)

BiosecurID: a multimodal biometric database

MULTIMODAL ANALYSIS: Informed content estimation and audio source separation

Learning Graph Representation of Person-specific Cognitive Processes from Audio-visual Behaviours for Automatic Personality Recognition

MediumVC: Any-to-any voice conversion using synthetic specific-speaker speeches as intermedium features

Audio-Visual Speech Recognition is Worth 32$\times$32$\times$8 Voxels

CTAL: Pre-training Cross-modal Transformer for Audio-and-Language Representations

Deep Variational Generative Models for Audio-visual Speech Separation

Audio-visual Multi-channel Integration and Recognition of Overlapped Speech

ACAV100M: Automatic Curation of Large-Scale Datasets for Audio-Visual Video Representation Learning

Towards a Generic Multimodal Architecture for Batch and Streaming Big Data Integration

ViNet: Pushing the limits of Visual Modality for Audio-Visual Saliency Prediction

Deep Music Retrieval for Fine-Grained Videos by Exploiting Cross-Modal-Encoded Voice-Overs

Multi-Modal Detection of Alzheimer's Disease from Speech and Text

Is Someone Speaking? Exploring Long-term Temporal Features for Audio-visual Active Speaker Detection

A Biologically Plausible Audio-Visual Integration Model for Continual Learning

Software Defined Media: Virtualization of Audio-Visual Services

Audio-Oriented Multimodal Machine Comprehension: Task, Dataset and Model

Cross-Modal Transformer-Based Neural Correction Models for Automatic Speech Recognition