多模态大模型 - arXivDaily 专题

2406.10448 2024-06-18 eess.AS cs.SD 83%

AVR: Synergizing Foundation Models for Audio-Visual Humor Detection

Sarthak Sharma, Orchid Chetia Phukan, Drishti Singh, Arun Balaji Buduru, Rajesh Sharma

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments Accepted to INTERSPEECH 2024 Show & Tell Demonstrations

2405.20773 2024-06-13 cs.CR cs.AI 83%

Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character

Siyuan Ma, Weidi Luo, Yu Wang, Xiaogeng Liu

专题命中音频语音多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.AI

2406.06163 2024-06-11 cs.CV 83%

Extending Segment Anything Model into Auditory and Temporal Dimensions for Audio-Visual Segmentation

Juhyeong Seon, Woobin Im, Sebin Lee, Jumin Lee, Sung-Eui Yoon

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted to ICIP 2024

2405.12983 2024-05-24 eess.AS cs.AI cs.CV cs.MM cs.SD 83%

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

Maxime Burchi, Krishna C. Puvvada, Jagadeesh Balam, Boris Ginsburg, Radu Timofte

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2311.12268 2024-04-25 cs.CV 83%

Boosting Audio-visual Zero-shot Learning with Large Language Models

Haoxing Chen, Yaohui Li, Yan Hong, Zizheng Huang, Zhuoer Xu, Zhangxuan Gu, Jun Lan, Huijia Zhu, Weiqiang Wang

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

2312.12870 2024-04-04 cs.CV 83%

The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective

Wenqi Jia, Miao Liu, Hao Jiang, Ishwarya Ananthabhotla, James M. Rehg, Vamsi Krishna Ithapu, Ruohan Gao

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

2404.00861 2024-04-02 eess.AS eess.IV 83%

Enhancing Real-World Active Speaker Detection with Multi-Modal Extraction Pre-Training

Ruijie Tao, Xinyuan Qian, Rohan Kumar Das, Xiaoxue Gao, Jiadong Wang, Haizhou Li

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments 10 pages

2403.02969 2024-03-26 cs.CV 83%

Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

Junwen He, Yifan Wang, Lijun Wang, Huchuan Lu, Jun-Yan He, Jin-Peng Lan, Bin Luo, Xuansong Xie

专题命中音频语音多模态：multi-modal(title)；multimodal(abstract)；MLLM(abstract)；分类 cs.CV

Comments CVPR 2024

2403.11074 2024-03-19 cs.CV cs.AI cs.MM cs.SD eess.AS 83%

Audio-Visual Segmentation via Unlabeled Frame Exploitation

Jinxiang Liu, Yikun Liu, Fei Zhang, Chen Ju, Ya Zhang, Yanfeng Wang

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by CVPR 2024

2402.16757 2024-02-27 cs.SD eess.AS 83%

Towards Environmental Preference Based Speech Enhancement For Individualised Multi-Modal Hearing Aids

Jasper Kirton-Wingate, Shafique Ahmed, Adeel Hussain, Mandar Gogate, Kia Dashtipour, Jen-Cheng Hou, Tassadaq Hussain, Yu Tsao, Amir Hussain

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments This has been submitted to the Trends in Hearing journal

2402.16394 2024-02-27 eess.AS cs.SD 83%

Audio-Visual Speech Enhancement in Noisy Environments via Emotion-Based Contextual Cues

Tassadaq Hussain, Kia Dashtipour, Yu Tsao, Amir Hussain

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 eess.AS

2305.18797 2024-02-14 cs.CV 83%

Learning Weakly Supervised Audio-Visual Violence Detection in Hyperbolic Space

Xiaogang Peng, Hao Wen, Yikai Luo, Xiao Zhou, Keyang Yu, Ping Yang, Zizhao Wu

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 11 pages, 12 figures, typos are fixed

2401.09150 2024-01-18 cs.CL 83%

Bridging Research and Readers: A Multi-Modal Automated Academic Papers Interpretation System

Feng Jiang, Kuang Wang, Haizhou Li

专题命中音频语音多模态：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CL

2401.05746 2024-01-12 cs.MM 83%

Cross-Modality and Within-Modality Regularization for Audio-Visual DeepFake Detection

Heqing Zou, Meng Shen, Yuchen Hu, Chen Chen, Eng Siong Chng, Deepu Rajan

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.MM

Comments Accepted by ICASSP 2024

2401.04023 2024-01-09 cs.CV cs.AI cs.LG cs.MM cs.SD eess.AS 83%

Efficient Multiscale Multimodal Bottleneck Transformer for Audio-Video Classification

Wentao Zhu

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Accepted by WACV 2024; well-formatted PDF is in https://drive.google.com/file/d/10Zo_ydJZFAm7YsxHDgTjhyc4dEJbW_dk/view?usp=sharing

2401.03468 2024-01-09 eess.AS cs.SD 83%

Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

Qiushi Zhu, Jie Zhang, Yu Gu, Yuchen Hu, Lirong Dai

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Comments Accepted by AAAI 2024

2309.05396 2023-12-27 cs.SD eess.AS 83%

SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpus

Haoxu Wang, Fan Yu, Xian Shi, Yuezhang Wang, Shiliang Zhang, Ming Li

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 eess.AS

Comments Accepted by ICASSP 2024

2312.12816 2023-12-21 cs.CV 83%

Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering

Zhangbin Li, Dan Guo, Jinxing Zhou, Jing Zhang, Meng Wang

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by AAAI-2024

2312.06720 2023-12-15 cs.CV 83%

Audio-Visual LLM for Video Understanding

Fangxun Shu, Lei Zhang, Hao Jiang, Cihang Xie

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

2311.17655 2023-11-30 cs.CV cs.AI cs.MM cs.SD eess.AS 83%

Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakes

Pavel Korshunov, Haolin Chen, Philip N. Garner, Sebastien Marcel

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 10 pages, 3 figures, 3 tables

2311.15080 2023-11-28 cs.CV cs.AI cs.LG cs.MM cs.SD eess.AS 83%

Weakly-Supervised Audio-Visual Segmentation

Shentong Mo, Bhiksha Raj

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

2311.04766 2023-11-14 cs.CV 83%

DualTalker: A Cross-Modal Dual Learning Approach for Speech-Driven 3D Facial Animation

Guinan Su, Yanwu Yang, Zhifeng Li

专题命中音频语音多模态：cross-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

2310.16477 2023-10-26 cs.CV 83%

Show from Tell: Audio-Visual Modelling in Clinical Settings

Jianbo Jiao, Mohammad Alsharid, Lior Drukker, Aris T. Papageorghiou, Andrew Zisserman, J. Alison Noble

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

2310.06383 2023-10-11 cs.AI 83%

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?

Siting Li, Chenzhuang Du, Yue Zhao, Yu Huang, Hang Zhao

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.AI

2310.05193 2023-10-10 cs.CV 83%

Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

Chenzhuang Du, Yue Zhao, Chonghua Liao, Jiacheng You, Jie Fu, Hang Zhao

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV

2309.17395 2023-10-02 cs.LG cs.SD eess.AS stat.ML 83%

AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition

Andrew Rouditchenko, Ronan Collobert, Tatiana Likhomanenko

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 eess.AS

Comments Under review

2212.04679 2023-09-21 cs.CV 83%

Motion and Context-Aware Audio-Visual Conditioned Video Prediction

Yating Xu, Conghui Hu, Gim Hee Lee

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments BMVC 2023

2309.10077 2023-09-20 cs.LG cs.AI 83%

GAME: Generalized deep learning model towards multimodal data integration for early screening of adolescent mental disorders

Zhicheng Du, Chenyao Jiang, Xi Yuan, Shiyao Zhai, Zhengyang Lei, Shuyue Ma, Yang Liu, Qihui Ye, Chufan Xiao, Qiming Huang, Ming Xu, Dongmei Yu, Peiwu Qin

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

2305.16406 2023-08-24 cs.CL 83%

Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech

Loukas Ilias, Dimitris Askounis

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL

Comments Knowledge-Based Systems

Journal ref Knowledge-Based Systems (Volume: 277, October 2023)

2308.10175 2023-08-22 cs.CV 83%

BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge

Chen Liu, Peike Li, Hu Zhang, Lincheng Li, Zi Huang, Dadong Wang, Xin Yu

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

1. 音频语音多模态 4549 篇

AVR: Synergizing Foundation Models for Audio-Visual Humor Detection

Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Character

Extending Segment Anything Model into Auditory and Temporal Dimensions for Audio-Visual Segmentation

Multilingual Audio-Visual Speech Recognition with Hybrid CTC/RNN-T Fast Conformer

Boosting Audio-visual Zero-shot Learning with Large Language Models

The Audio-Visual Conversational Graph: From an Egocentric-Exocentric Perspective

Enhancing Real-World Active Speaker Detection with Multi-Modal Extraction Pre-Training

Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

Audio-Visual Segmentation via Unlabeled Frame Exploitation

Towards Environmental Preference Based Speech Enhancement For Individualised Multi-Modal Hearing Aids

Audio-Visual Speech Enhancement in Noisy Environments via Emotion-Based Contextual Cues

Learning Weakly Supervised Audio-Visual Violence Detection in Hyperbolic Space

Bridging Research and Readers: A Multi-Modal Automated Academic Papers Interpretation System

Cross-Modality and Within-Modality Regularization for Audio-Visual DeepFake Detection

Efficient Multiscale Multimodal Bottleneck Transformer for Audio-Video Classification

Multichannel AV-wav2vec2: A Framework for Learning Multichannel Multi-Modal Speech Representation

SlideSpeech: A Large-Scale Slide-Enriched Audio-Visual Corpus

Object-aware Adaptive-Positivity Learning for Audio-Visual Question Answering

Audio-Visual LLM for Video Understanding

Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakes

Weakly-Supervised Audio-Visual Segmentation

DualTalker: A Cross-Modal Dual Learning Approach for Speech-Driven 3D Facial Animation

Show from Tell: Audio-Visual Modelling in Clinical Settings

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?

Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

AV-CPL: Continuous Pseudo-Labeling for Audio-Visual Speech Recognition

Motion and Context-Aware Audio-Visual Conditioned Video Prediction

GAME: Generalized deep learning model towards multimodal data integration for early screening of adolescent mental disorders

Context-aware attention layers coupled with optimal transport domain adaptation and multimodal fusion methods for recognizing dementia from spontaneous speech

BAVS: Bootstrapping Audio-Visual Segmentation by Integrating Foundation Knowledge