多模态大模型 - arXivDaily 专题

2410.23861 2024-11-01 cs.CL cs.MM cs.SD eess.AS 82%

Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Models

Hao Yang, Lizhen Qu, Ehsan Shareghi, Gholamreza Haffari

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

2410.22803 2024-10-31 cs.SD cs.AI cs.LG cs.MM eess.AS 82%

DOA-Aware Audio-Visual Self-Supervised Learning for Sound Event Localization and Detection

Yoto Fujita, Yoshiaki Bando, Keisuke Imoto, Masaki Onishi, Kazuyoshi Yoshii

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Accepted to APSIPA2023

2403.06734 2024-10-27 cs.AI cs.CL cs.CV 82%

Real-Time Multimodal Cognitive Assistant for Emergency Medical Services

Keshara Weerasinghe, Saahith Janapati, Xueren Ge, Sion Kim, Sneha Iyer, John A. Stankovic, Homa Alemzadeh

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments This work has been submitted to the IEEE for possible publication

2307.05435 2024-10-23 cs.LG 82%

One-Versus-Others Attention: Scalable Multimodal Integration for Biomedical Data

Michal Golovanevsky, Eva Schiller, Akira Nair, Eric Han, Ritambhara Singh, Carsten Eickhoff

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)

2410.13445 2024-10-18 cs.CL cs.AI cs.LG eess.AS 82%

Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR

Abhishek Gupta, Amruta Parulekar, Sameep Chattopadhyay, Preethi Jyothi

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

2410.11582 2024-10-16 cs.CV cs.AI cs.LG cs.MM 82%

On-the-fly Modulation for Balanced Multimodal Learning

Yake Wei, Di Hu, Henghui Du, Ji-Rong Wen

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by T-PAMI 2024

2408.06753 2024-10-15 cs.CV cs.MM cs.SD eess.AS 82%

Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistencies

Marcella Astrid, Enjie Ghorbel, Djamila Aouada

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted in BMVC 2024

2409.19575 2024-10-01 cs.SD cs.CL cs.MM eess.AS 82%

Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective

Chen Chen, Xiaolou Li, Zehua Liu, Lantian Li, Dong Wang

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted by ISCSLP2024

2409.19132 2024-10-01 cs.MM cs.CV cs.LG cs.SD eess.AS 82%

From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation

Kun Su, Xiulong Liu, Eli Shlizerman

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by ICML 2024

2409.18971 2024-10-01 cs.MM cs.AI cs.SD eess.AS 82%

Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better

Mengying Ge, Mingyang Li, Dongkai Tang, Pengbo Li, Kuo Liu, Shuhao Deng, Songbai Pu, Long Liu, Yang Song, Tao Zhang

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.AI、cs.MM、eess.AS

2406.18088 2024-09-25 cs.CL cs.AI cs.SD eess.AS 82%

LLM-Driven Multimodal Opinion Expression Identification

Bonian Jia, Huiyao Chen, Yueheng Sun, Meishan Zhang, Min Zhang

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments 5 pages, 3 Figures, Accept by Interspeech 2024

Journal ref Proceedings of Interspeech 2024

2409.14340 2024-09-24 cs.CV cs.LG cs.MM cs.SD eess.AS 82%

Self-Supervised Audio-Visual Soundscape Stylization

Tingle Li, Renhao Wang, Po-Yao Huang, Andrew Owens, Gopala Anumanchipalli

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments ECCV 2024

2409.11729 2024-09-19 cs.MM cs.CV cs.SD eess.AS 82%

DETECLAP: Enhancing Audio-Visual Representation Learning with Object Information

Shota Nakada, Taichi Nishimura, Hokuto Munakata, Masayoshi Kondo, Tatsuya Komatsu

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments under review

2409.09611 2024-09-17 cs.CV cs.AI cs.LG cs.SD eess.AS 82%

Integrating Audio Narrations to Strengthen Domain Generalization in Multimodal First-Person Action Recognition

Cagri Gungor, Adriana Kovashka

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、eess.AS

2409.06709 2024-09-12 cs.MM cs.AI cs.SD eess.AS 82%

Unveiling Visual Biases in Audio-Visual Localization Benchmarks

Liangyu Chen, Zihao Yue, Boshen Xu, Qin Jin

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.AI、cs.MM、eess.AS

Comments Accepted by ECCV24 AVGenL Workshop

2409.02239 2024-09-06 cs.SD cs.AI cs.CL eess.AS 82%

Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASR

Xugang Lu, Peng Shen, Yu Tsao, Hisashi Kawai

专题命中音频语音多模态：cross-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted to IEEE SLT 2024

2406.14176 2024-08-20 cs.SD cs.AI cs.MM eess.AS 82%

A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection

Kyungbok Lee, You Zhang, Zhiyao Duan

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.AI、cs.MM、eess.AS

2408.01337 2024-08-05 cs.SD cs.CL cs.LG cs.MM eess.AS 82%

MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models

Benno Weck, Ilaria Manco, Emmanouil Benetos, Elio Quinton, George Fazekas, Dmitry Bogdanov

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted at ISMIR 2024. Data: https://doi.org/10.5281/zenodo.12709974 Code: https://github.com/mulab-mir/muchomusic Supplementary material: https://mulab-mir.github.io/muchomusic

2303.17395 2024-07-22 eess.AS cs.CL cs.MM cs.SD 82%

WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research

Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D. Plumbley, Yuexian Zou, Wenwu Wang

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted to TASLP

2407.11650 2024-07-18 cs.CV cs.MM cs.SD eess.AS 82%

Statistics-aware Audio-visual Deepfake Detector

Marcella Astrid, Enjie Ghorbel, Djamila Aouada

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted in ICIP 2024

2407.11492 2024-07-17 cs.SD cs.CL cs.MM eess.AS 82%

MMSD-Net: Towards Multi-modal Stuttering Detection

Liangyu Nie, Sudarsana Reddy Kadiri, Ruchit Agrawal

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted at INTERSPEECH 2024

2407.08130 2024-07-12 cs.MM cs.CV cs.SD eess.AS 82%

Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning

Wenrui Li, Penghong Wang, Ruiqin Xiong, Xiaopeng Fan

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by TIP

2407.07825 2024-07-11 cs.SD cs.CV cs.MM eess.AS 82%

RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement

Honglie Chen, Rodrigo Mira, Stavros Petridis, Maja Pantic

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Interspeech 2024

2407.02004 2024-07-08 cs.CV cs.AI cs.SD eess.AS 82%

SAVE: Segment Audio-Visual Easy way using Segment Anything Model

Khanh-Binh Nguyen, Chae Jung Park

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、eess.AS

2406.15754 2024-06-25 cs.CV cs.CL cs.LG cs.SD eess.AS 82%

Multimodal Segmentation for Vocal Tract Modeling

Rishi Jain, Bohan Yu, Peter Wu, Tejas Prabhune, Gopala Anumanchipalli

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Interspeech 2024

2406.13384 2024-06-21 cs.SD cs.CV cs.MM eess.AS 82%

Straight Through Gumbel Softmax Estimator based Bimodal Neural Architecture Search for Audio-Visual Deepfake Detection

Aravinda Reddy PN, Raghavendra Ramachandra, Krothapalli Sreenivasa Rao, Pabitra Mitra, Vinod Rathod

专题命中音频语音多模态：audio-visual(title)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

2406.09618 2024-06-17 cs.CL cs.AI cs.IR cs.SD eess.AS 82%

Multi-Modal Retrieval For Large Language Model Based Speech Recognition

Jari Kolehmainen, Aditya Gourav, Prashanth Gurunath Shivakumar, Yile Gu, Ankur Gandhe, Ariya Rastrow, Grant Strimel, Ivan Bulyko

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

2406.09569 2024-06-17 cs.CL cs.AI cs.SD eess.AS 82%

Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time

Frank Seide, Morrie Doulaty, Yangyang Shi, Yashesh Gaur, Junteng Jia, Chunyang Wu

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

2406.09156 2024-06-14 cs.LG cs.CV cs.MM cs.SD eess.AS 82%

Towards Multilingual Audio-Visual Question Answering

Orchid Chetia Phukan, Priyabrata Mallick, Swarup Ranjan Behera, Aalekhya Satya Narayani, Arun Balaji Buduru, Rajesh Sharma

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to Interspeech 2024

2406.06438 2024-06-11 cs.CL cs.CV cs.HC cs.LG cs.SD eess.AS 82%

Multimodal Contextualized Semantic Parsing from Speech

Jordan Voas, Raymond Mooney, David Harwath

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、eess.AS

Comments 10 Pages, 3 figures, ACL 2024 Main

1. 音频语音多模态 4557 篇

Audio Is the Achilles' Heel: Red Teaming Audio Large Multimodal Models

DOA-Aware Audio-Visual Self-Supervised Learning for Sound Event Localization and Detection

Real-Time Multimodal Cognitive Assistant for Emergency Medical Services

One-Versus-Others Attention: Scalable Multimodal Integration for Biomedical Data

Parameter-efficient Adaptation of Multilingual Multimodal Models for Low-resource ASR

On-the-fly Modulation for Balanced Multimodal Learning

Detecting Audio-Visual Deepfakes with Fine-Grained Inconsistencies

Quantitative Analysis of Audio-Visual Tasks: An Information-Theoretic Perspective

From Vision to Audio and Beyond: A Unified Model for Audio-Visual Representation and Generation

Early Joint Learning of Emotion Information Makes MultiModal Model Understand You Better

LLM-Driven Multimodal Opinion Expression Identification

Self-Supervised Audio-Visual Soundscape Stylization

DETECLAP: Enhancing Audio-Visual Representation Learning with Object Information

Integrating Audio Narrations to Strengthen Domain Generalization in Multimodal First-Person Action Recognition

Unveiling Visual Biases in Audio-Visual Localization Benchmarks

Temporal Order Preserved Optimal Transport-based Cross-modal Knowledge Transfer Learning for ASR

A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection

MuChoMusic: Evaluating Music Understanding in Multimodal Audio-Language Models

WavCaps: A ChatGPT-Assisted Weakly-Labelled Audio Captioning Dataset for Audio-Language Multimodal Research

Statistics-aware Audio-visual Deepfake Detector

MMSD-Net: Towards Multi-modal Stuttering Detection

Spiking Tucker Fusion Transformer for Audio-Visual Zero-Shot Learning

RT-LA-VocE: Real-Time Low-SNR Audio-Visual Speech Enhancement

SAVE: Segment Audio-Visual Easy way using Segment Anything Model

Multimodal Segmentation for Vocal Tract Modeling

Straight Through Gumbel Softmax Estimator based Bimodal Neural Architecture Search for Audio-Visual Deepfake Detection

Multi-Modal Retrieval For Large Language Model Based Speech Recognition

Speech ReaLLM -- Real-time Streaming Speech Recognition with Multimodal LLMs by Teaching the Flow of Time

Towards Multilingual Audio-Visual Question Answering

Multimodal Contextualized Semantic Parsing from Speech