多模态大模型 - arXivDaily 专题

2407.13676 2024-07-19 cs.MM cs.CV cs.SD eess.AS 85%

Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment

Arda Senocak, Hyeonggon Ryu, Junsik Kim, Tae-Hyun Oh, Hanspeter Pfister, Joon Son Chung

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Journal Extension of ICCV 2023 paper (arXiV:2309.10724). Code is available at https://github.com/kaistmm/SSLalignment

2407.13095 2024-07-19 cs.CV cs.LG cs.MM cs.SD eess.AS 85%

Audio-visual Generalized Zero-shot Learning the Easy Way

Shentong Mo, Pedro Morgado

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2407.08126 2024-07-12 cs.AI cs.CV cs.MM 85%

Label-anticipated Event Disentanglement for Audio-Visual Video Parsing

Jinxing Zhou, Dan Guo, Yuxin Mao, Yiran Zhong, Xiaojun Chang, Meng Wang

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by ECCV2024

2407.01851 2024-07-04 cs.CV cs.AI cs.LG eess.AS 85%

Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time

Sanjoy Chowdhury, Sayan Nag, Subhrajyoti Dasgupta, Jun Chen, Mohamed Elhoseiny, Ruohan Gao, Dinesh Manocha

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、eess.AS

Comments Accepted at ECCV 2024

2401.09759 2024-07-03 cs.CV cs.MM cs.SD eess.AS 85%

SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition

Hao Wang, Shuhei Kurita, Shuichiro Shimizu, Daisuke Kawahara

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 3rd Workshop on Advances in Language and Vision Research (ALVR 2024)

2406.02951 2024-06-06 cs.CV cs.MM cs.SD eess.AS 85%

AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection

Trevine Oorloff, Surya Koppisetti, Nicolò Bonettini, Divyaraj Solanki, Ben Colman, Yaser Yacoob, Ali Shahriyari, Gaurav Bharaj

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to CVPR 2024

2405.14598 2024-05-27 cs.CV cs.LG cs.MM cs.SD eess.AS 85%

Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation

Shiqi Yang, Zhi Zhong, Mengjie Zhao, Shusuke Takahashi, Masato Ishii, Takashi Shibuya, Yuki Mitsufuji

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 10 pages

2404.13789 2024-04-24 cs.SD cs.AI cs.IR cs.MM eess.AS 85%

Anchor-aware Deep Metric Learning for Audio-visual Retrieval

Donghuo Zeng, Yanan Wang, Kazushi Ikeda, Yi Yu

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments 9 pages, 5 figures. Accepted by ACM ICMR 2024

2302.10035 2024-04-11 cs.CV cs.AI cs.MM 85%

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

Xiao Wang, Guangyao Chen, Guangwu Qian, Pengcheng Gao, Xiao-Yong Wei, Yaowei Wang, Yonghong Tian, Wen Gao

专题命中音频语音多模态：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by Machine Intelligence Research (MIR)

2403.04245 2024-03-08 cs.SD cs.CV cs.LG cs.MM eess.AS 85%

A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition

Yusheng Dai, Hang Chen, Jun Du, Ruoyu Wang, Shihao Chen, Jiefeng Ma, Haotian Wang, Chin-Hui Lee

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments the paper is accepted by CVPR2024

2312.01017 2023-12-05 cs.CV cs.AI cs.LG cs.MM cs.SD 85%

Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling

Shentong Mo, Pedro Morgado

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

2310.20446 2023-11-01 cs.SD cs.CV cs.MM eess.AS 85%

LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

Yuxin Ye, Wenming Yang, Yapeng Tian

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by WACV2024

2310.13451 2023-10-23 cs.SD cs.CV cs.IR cs.MM eess.AS 85%

Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval

Donghuo Zeng, Kazushi Ikeda

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 8 pages, 6 figures

2309.16569 2023-09-29 cs.SD cs.CV cs.MM eess.AS 85%

Audio-Visual Speaker Verification via Joint Cross-Attention

R. Gnana Praveen, Jahangir Alam

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2309.06728 2023-09-14 cs.CV cs.MM cs.SD eess.AS 85%

Leveraging Foundation models for Unsupervised Audio-Visual Segmentation

Swapnil Bhosale, Haosen Yang, Diptesh Kanojia, Xiatian Zhu

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2308.13421 2023-08-28 cs.CV cs.MM cs.SD eess.AS 85%

Exploiting Diverse Feature for Multimodal Sentiment Analysis

Jia Li, Wei Qian, Kun Li, Qi Li, Dan Guo, Meng Wang

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

2308.09322 2023-08-21 cs.CV cs.AI cs.MM 85%

Audio-Visual Glance Network for Efficient Video Recognition

Muhammad Adi Nugroho, Sangmin Woo, Sumin Lee, Changick Kim

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments ICCV 2023

2307.15097 2023-07-31 cs.CL cs.LG cs.MM eess.AS 85%

Cascaded Cross-Modal Transformer for Request and Complaint Detection

Nicolae-Catalin Ristea, Radu Tudor Ionescu

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted at ACMMM 2023

2307.13236 2023-07-26 cs.SD cs.CV cs.LG cs.MM eess.AS 85%

Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation

Jinxiang Liu, Chen Ju, Chaofan Ma, Yanfeng Wang, Yu Wang, Ya Zhang

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments arXiv admin note: text overlap with arXiv:2305.11019

2306.10684 2023-06-21 cs.SD cs.CV cs.MM eess.AS 85%

Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding

Zengjie Song, Zhaoxiang Zhang

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted to IEEE Transactions on Neural Networks and Learning Systems (T-NNLS)

2306.10567 2023-06-21 eess.AS cs.CV cs.MM cs.SD 85%

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

Yuchen Hu, Chen Chen, Ruizhe Li, Heqing Zou, Eng Siong Chng

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments 14 pages, 5 figures, Accepted by ACL 2023

2210.16428 2023-05-30 eess.AS cs.AI cs.MM cs.SD 85%

Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention

Xubo Liu, Qiushi Huang, Xinhao Mei, Haohe Liu, Qiuqiang Kong, Jianyuan Sun, Shengchen Li, Tom Ko, Yu Zhang, Lilian H. Tang, Mark D. Plumbley, Volkan Kılıç, Wenwu Wang

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.AI、cs.MM、eess.AS

Comments INTERSPEECH 2023

2305.01836 2023-05-04 cs.CV cs.LG cs.MM cs.SD eess.AS 85%

AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation

Shentong Mo, Yapeng Tian

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

2303.12930 2023-03-27 cs.CV cs.MM cs.SD eess.AS 85%

Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline

Tiantian Geng, Teng Wang, Jinming Duan, Runmin Cong, Feng Zheng

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by CVPR2023

2303.13471 2023-03-24 cs.CV cs.MM cs.SD eess.AS 85%

Egocentric Audio-Visual Object Localization

Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by CVPR 2023

2303.08536 2023-03-21 cs.MM cs.CV cs.LG cs.SD eess.AS 85%

Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring

Joanna Hong, Minsu Kim, Jeongsoo Choi, Yong Man Ro

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at CVPR 2023. Implementation available: https://github.com/joannahong/AV-RelScore

2302.13033 2023-02-28 cs.SD cs.CV cs.MM eess.AS 85%

Speaker Recognition in Realistic Scenario Using Multimodal Data

Saqlain Hussain Shah, Muhammad Saad Saeed, Shah Nawaz, Muhammad Haroon Yousaf

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted at the International Conference on Artificial Intelligence (ICAI'2023)

2212.01083 2023-02-28 cs.CV cs.CL cs.SD eess.AS 85%

Cross-Modal Mutual Learning for Cued Speech Recognition

Lei Liu, Li Liu

专题命中音频语音多模态：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、eess.AS

Comments Accepted to ICASSP2023

2210.07940 2022-10-17 cs.CV cs.AI cs.CL cs.LG 85%

AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments

Sudipta Paul, Amit K. Roy-Chowdhury, Anoop Cherian

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at NeurIPS 2022

2210.07055 2022-10-14 cs.CV cs.LG cs.MM cs.SD eess.AS 85%

Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors

Vladimir Iashin, Weidi Xie, Esa Rahtu, Andrew Zisserman

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted as a spotlight presentation for the BMVC 2022. Code: https://github.com/v-iashin/SparseSync Project page: https://v-iashin.github.io/SparseSync

1. 音频语音多模态 4549 篇

Aligning Sight and Sound: Advanced Sound Source Localization Through Audio-Visual Alignment

Audio-visual Generalized Zero-shot Learning the Easy Way

Label-anticipated Event Disentanglement for Audio-Visual Video Parsing

Meerkat: Audio-Visual Large Language Model for Grounding in Space and Time

SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition

AVFF: Audio-Visual Feature Fusion for Video Deepfake Detection

Visual Echoes: A Simple Unified Transformer for Audio-Visual Generation

Anchor-aware Deep Metric Learning for Audio-visual Retrieval

Large-scale Multi-Modal Pre-trained Models: A Comprehensive Survey

A Study of Dropout-Induced Modality Bias on Robustness to Missing Video Frames for Audio-Visual Speech Recognition

Unveiling the Power of Audio-Visual Early Fusion Transformers with Dense Interactions through Masked Modeling

LAVSS: Location-Guided Audio-Visual Spatial Audio Separation

Two-Stage Triplet Loss Training with Curriculum Augmentation for Audio-Visual Retrieval

Audio-Visual Speaker Verification via Joint Cross-Attention

Leveraging Foundation models for Unsupervised Audio-Visual Segmentation

Exploiting Diverse Feature for Multimodal Sentiment Analysis

Audio-Visual Glance Network for Efficient Video Recognition

Cascaded Cross-Modal Transformer for Request and Complaint Detection

Audio-aware Query-enhanced Transformer for Audio-Visual Segmentation

Visually-Guided Sound Source Separation with Audio-Visual Predictive Coding

MIR-GAN: Refining Frame-Level Modality-Invariant Representations with Adversarial Network for Audio-Visual Speech Recognition

Visually-Aware Audio Captioning With Adaptive Audio-Visual Attention

AV-SAM: Segment Anything Model Meets Audio-Visual Localization and Segmentation

Dense-Localizing Audio-Visual Events in Untrimmed Videos: A Large-Scale Benchmark and Baseline

Egocentric Audio-Visual Object Localization

Watch or Listen: Robust Audio-Visual Speech Recognition with Visual Corruption Modeling and Reliability Scoring

Speaker Recognition in Realistic Scenario Using Multimodal Data

Cross-Modal Mutual Learning for Cued Speech Recognition

AVLEN: Audio-Visual-Language Embodied Navigation in 3D Environments

Sparse in Space and Time: Audio-visual Synchronisation with Trainable Selectors