多模态大模型

AnnoTheia: A Semi-Automatic Annotation Toolkit for Audio-Visual Speech Technologies

José-M. Acosta-Triana, David Gimeno-Gómez, Carlos-D. Martínez-Hinarejos

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted at the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING)

Looking for a better fit? An Incremental Learning Multimodal Object Referencing Framework adapting to Individual Drivers

Amr Gomaa, Guillermo Reyes, Michael Feld, Antonio Krüger

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted for publication in the Proceedings of the 29th International Conference on Intelligent User Interfaces (IUI'24), March 18--21, 2024, in Greenville, SC, USA

Bootstrapping Audio-Visual Segmentation by Strengthening Audio Cues

Tianxiang Chen, Zhentao Tan, Tao Gong, Qi Chu, Yue Wu, Bin Liu, Le Lu, Jieping Ye, Nenghai Yu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Enlighten-Your-Voice: When Multimodal Meets Zero-shot Low-light Image Enhancement

Xiaofeng Zhang, Zishan Xu, Hao Tang, Chaochen Gu, Wei Chen, Shanying Zhu, Xinping Guan

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Comments It needs revised

Enhanced Sound Event Localization and Detection in Real 360-degree audio-visual soundscapes

Adrian S. Roman, Baladithya Balamurugan, Rithik Pothuganti

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、eess.AS

TDFNet: An Efficient Audio-Visual Speech Separation Model with Top-down Fusion

Samuel Pegg, Kai Li, Xiaolin Hu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、eess.AS

Journal ref 2023 13th International Conference on Information Science and Technology (ICIST), Cairo, Egypt, 2023, pp. 243-252

Look, Listen and Recognise: Character-Aware Audio-Visual Subtitling

Bruno Korbar, Jaesung Huh, Andrew Zisserman

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted for publication in ICASSP 2024

TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus

Rafael Ferreira, Diogo Tavares, Diogo Silva, Rodrigo Valério, João Bordalo, Inês Simões, Vasco Ramos, David Semedo, João Magalhães

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

LCB-net: Long-Context Biasing for Audio-Visual Speech Recognition

Fan Yu, Haoxu Wang, Xian Shi, Shiliang Zhang

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted by ICASPP 2024

HCAM -- Hierarchical Cross Attention Model for Multi-modal Emotion Recognition

Soumya Dutta, Sriram Ganapathy

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、eess.AS

Comments 11 pages, 6 figures

Bridging Modalities: Knowledge Distillation and Masked Training for Translating Multi-Modal Emotion Recognition to Uni-Modal, Speech-Only Emotion Recognition

Muhammad Muaz, Nathan Paull, Jahnavi Malagavalli

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.AI、eess.AS

MARRS: Multimodal Reference Resolution System

Halim Cagri Ates, Shruti Bhargava, Site Li, Jiarui Lu, Siddhardha Maddula, Joel Ruben Antony Moniz, Anil Kumar Nalamalapu, Roman Hoang Nguyen, Melis Ozyildirim, Alkesh Patel, Dhivya Piraviperumal, Vincent Renkens, Ankit Samal, Thy Tran, Bo-Hsiang Tseng, Hong Yu, Yuan Zhang, Rong Zou

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Sixth Workshop on Computational Models of Reference, Anaphora and Coreference (CRAC 2023)

Churn Prediction via Multimodal Fusion Learning:Integrating Customer Financial Literacy, Voice, and Behavioral Data

David Hason Rudd, Huan Huo, Md Rafiqul Islam, Guandong Xu

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Audio-visual fine-tuning of audio-only ASR models

Avner May, Dmitriy Serdyuk, Ankit Parag Shah, Otavio Braga, Olivier Siohan

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.AI、eess.AS

Stylebook: Content-Dependent Speaking Style Modeling for Any-to-Any Voice Conversion using Only Speech Data

Hyungseob Lim, Kyungguen Byun, Sunkuk Moon, Erik Visser

专题命中 音频语音多模态 ：any-to-any(title,abstract)；分类 cs.AI、eess.AS

Comments 5 pages, 2 figures, 2 tables

Design, construction and evaluation of emotional multimodal pathological speech database

Ting Zhu, Shufei Duan, Huizhi Liang, Wei Zhang

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI、eess.AS

WhisBERT: Multimodal Text-Audio Language Modeling on 100M Words

Lukas Wolf, Greta Tuckute, Klemen Kotar, Eghbal Hosseini, Tamar Regev, Ethan Wilcox, Alex Warstadt

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments Published at the BabyLM Challenge, a shared task co-sponsored by CMCL 2023 and CoNLL 2023, hosted by EMNLP 2023

Cooperative Dual Attention for Audio-Visual Speech Enhancement with Facial Cues

Feixiang Wang, Shuang Yang, Shiguang Shan, Xilin Chen

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted to BMVC 2023 15 pages, 2 figures

STHG: Spatial-Temporal Heterogeneous Graph Learning for Advanced Audio-Visual Diarization

Kyle Min

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Validation report for the Ego4D challenge at CVPR 2023

Scenario-Aware Audio-Visual TF-GridNet for Target Speech Extraction

Zexu Pan, Gordon Wichern, Yoshiki Masuyama, Francois G. Germain, Sameer Khurana, Chiori Hori, Jonathan Le Roux

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments Accepted by ASRU 2023

Seeing Through the Conversation: Audio-Visual Speech Separation based on Diffusion Model

Suyeon Lee, Chaeyoung Jung, Youngjoon Jang, Jaehun Kim, Joon Son Chung

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Project page with demo: https://mm.kaist.ac.kr/projects/avdiffuss/

Intel Labs at Ego4D Challenge 2022: A Better Baseline for Audio-Visual Diarization

Kyle Min

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Validation report for the Ego4D challenge at ECCV 2022

Intuitive Multilingual Audio-Visual Speech Recognition with a Single-Trained Model

Joanna Hong, Se Jin Park, Yong Man Ro

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.MM、eess.AS

Comments EMNLP 2023 Findings

Deep Video Inpainting Guided by Audio-Visual Self-Supervision

Kyuyeon Kim, Junsik Jung, Woo Jae Kim, Sung-Eui Yoon

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、eess.AS

Comments Accepted at ICASSP 2022

Audio-Visual Neural Syntax Acquisition

Cheng-I Jeff Lai, Freda Shi, Puyuan Peng, Yoon Kim, Kevin Gimpel, Shiyu Chang, Yung-Sung Chuang, Saurabhchand Bhati, David Cox, David Harwath, Yang Zhang, Karen Livescu, James Glass

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CL、eess.AS

Multimodal Speech Recognition for Language-Guided Embodied Agents

Allen Chang, Xiaoyuan Zhu, Aarav Monga, Seoho Ahn, Tejas Srinivasan, Jesse Thomason

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL、eess.AS

Comments 5 pages, 5 figures

Journal ref Proceedings of Interspeech 2023, 1608-1612

Multi-Resolution Audio-Visual Feature Fusion for Temporal Action Localization

Edward Fish, Jon Weinbren, Andrew Gilbert

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV、cs.MM

Comments Under Review

Abusing Images and Sounds for Indirect Instruction Injection in Multi-Modal LLMs

Eugene Bagdasaryan, Tsung-Yin Hsieh, Ben Nassi, Vitaly Shmatikov

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CL、cs.AI

Sarcasm in Sight and Sound: Benchmarking and Expansion to Improve Multimodal Sarcasm Detection

Swapnil Bhosale, Abhra Chaudhuri, Alex Lee Robert Williams, Divyank Tiwari, Anjan Dutta, Xiatian Zhu, Pushpak Bhattacharyya, Diptesh Kanojia

专题命中 音频语音多模态 ：multimodal(title)；multi-modal(abstract)；分类 cs.CL、cs.AI

Cross-Modal Multi-Tasking for Speech-to-Text Translation via Hard Parameter Sharing

Brian Yan, Xuankai Chang, Antonios Anastasopoulos, Yuya Fujita, Shinji Watanabe

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 cs.CL、eess.AS