多模态大模型 - arXivDaily 专题

2503.06108 2025-03-11 cs.SD cs.AI 83%

Multi-modal expressive personality recognition in data non-ideal audiovisual based on multi-scale feature enhancement and modal augment

Weixuan Kong, Jinpeng Yu, Zijun Li, Hanwei Liu, Jiqing Qu, Hui Xiao, Xuefeng Li

专题命中音频语音多模态：multi-modal(title)；multimodal(abstract)；audio-visual(abstract)；分类 cs.AI

2404.12020 2025-03-06 cs.CV 83%

Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering

Jie Ma, Min Hu, Pinghui Wang, Wangchun Sun, Lingyun Song, Hongbin Pei, Jun Liu, Youtian Du

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by NeurIPS 2024

2502.11858 2025-03-04 cs.SD cs.CV 83%

Rethinking Audio-Visual Adversarial Vulnerability from Temporal and Modality Perspectives

Zeliang Zhang, Susan Liang, Daiki Shimada, Chenliang Xu

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments Accepted by ICLR 2025

2502.11775 2025-02-18 cs.CV 83%

video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model

Guangzhi Sun, Yudong Yang, Jimin Zhuang, Changli Tang, Yixuan Li, Wei Li, Zejun MA, Chao Zhang

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

2502.05758 2025-02-11 eess.AS 83%

Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation

Jing-Xuan Zhang, Tingzhi Mao, Longjiang Guo, Jin Li, Lichen Zhang

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 eess.AS

Comments accepted to ESWA journal

2502.04976 2025-02-10 cs.MM 83%

Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmark

Han Zhang, Zixiang Meng, Meng Luo, Hong Han, Lizi Liao, Erik Cambria, Hao Fei

专题命中音频语音多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.MM

Comments Accepted by TheWebConf (WWW) 2025

2502.03005 2025-02-06 cs.CV cs.LG 83%

Driver Assistance System Based on Multimodal Data Hazard Detection

Long Zhouxiang, Ovanes Petrosian

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

2501.01518 2025-01-06 eess.AS cs.SD eess.SP 83%

Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation

Akam Rahimi, Triantafyllos Afouras, Andrew Zisserman

专题命中音频语音多模态：multi-modal(title,abstract)；audio-visual(abstract)；分类 eess.AS

Journal ref 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

2412.17907 2024-12-25 cs.HC cs.CL cs.CV cs.LG cs.MM cs.SD eess.AS 83%

A Multimodal Emotion Recognition System: Integrating Facial Expressions, Body Movement, Speech, and Spoken Language

Kris Kraack

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments 10 pages, 6 figures, 3 tables

2412.16530 2024-12-24 cs.SD cs.CL cs.CV cs.MM eess.AS 83%

Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translation

Lucas Goncalves, Prashant Mathur, Xing Niu, Brady Houston, Chandrashekhar Lavania, Srikanth Vishnubhotla, Lijia Sun, Anthony Ferritto

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted at ICASSP, 4 pages

2407.06125 2024-12-03 cs.HC cs.AI 83%

Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalities

Chayan Tank, Sarthak Pol, Vinayak Katoch, Shaina Mehta, Avinash Anand, Rajiv Ratn Shah

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.AI

Comments 12 pages, 9 figures, 9 tables

2412.00049 2024-12-03 cs.MM cs.AI cs.CV cs.SD eess.AS 83%

A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning

Luis Vilaca, Yi Yu, Paula Vinan

专题命中音频语音多模态：audio-visual(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments arXiv admin note: text overlap with arXiv:2202.13673

2411.08885 2024-11-18 cs.MM cs.AI cs.CV cs.SD eess.AS 83%

Enhancing Lie Detection Accuracy: A Comparative Study of Classic ML, CNN, and GCN Models using Audio-Visual Features

Abdelrahman Abdelwahab, Akshaj Vishnubhatla, Ayaan Vaswani, Advait Bharathulwar, Arnav Kommaraju

专题命中音频语音多模态：audio-visual(title)；multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 11 pages, 18 figures

2411.04933 2024-11-12 cs.CV 83%

SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering

Tianyu Yang, Yiyang Nan, Lisen Dai, Zhenwen Liang, Yapeng Tian, Xiangliang Zhang

专题命中音频语音多模态：audio-visual(title,abstract)；multi-modal(abstract)；分类 cs.CV

Comments EMNLP 2024

2411.05903 2024-11-12 cs.LG cs.AI cs.CL cs.CV cs.SD eess.AS 83%

Towards Multi-Modal Mastery: A 4.5B Parameter Truly Multi-Modal Small Language Model

Ben Koska, Mojmír Horváth

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2410.07463 2024-11-12 cs.CV 83%

Language-Guided Joint Audio-Visual Editing via One-Shot Adaptation

Susan Liang, Chao Huang, Yapeng Tian, Anurag Kumar, Chenliang Xu

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments ACCV 2024

2410.12787 2024-10-17 cs.CV 83%

The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio

Sicong Leng, Yun Xing, Zesen Cheng, Yang Zhou, Hang Zhang, Xin Li, Deli Zhao, Shijian Lu, Chunyan Miao, Lidong Bing

专题命中音频语音多模态：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Project Page: cmm-damovl.site

2407.05358 2024-10-01 cs.CV 83%

CPM: Class-conditional Prompting Machine for Audio-visual Segmentation

Yuanhong Chen, Chong Wang, Yuyuan Liu, Hu Wang, Gustavo Carneiro

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

2311.09580 2024-09-27 cs.CL 83%

MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts

Haofei Yu, Zhengyang Qi, Lawrence Jang, Ruslan Salakhutdinov, Louis-Philippe Morency, Paul Pu Liang

专题命中音频语音多模态：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL

2409.02243 2024-09-05 cs.CV 83%

A Novel Audio-Visual Information Fusion System for Mental Disorders Detection

Yichun Li, Shuanglin Li, Syed Mohsen Naqvi

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments 27th International Conference on Information (FUSION)

2404.10904 2024-09-05 cs.CV 83%

Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition

Marah Halawa, Florian Blume, Pia Bideau, Martin Maier, Rasha Abdel Rahman, Olaf Hellwich

专题命中音频语音多模态：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments The paper will appear in the CVPR 2024 workshops proceedings

Journal ref Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, 2024, pp. 4604-4614

2408.10246 2024-08-21 cs.CV cs.AI cs.CL eess.AS 83%

VyAnG-Net: A Novel Multi-Modal Sarcasm Recognition Model by Uncovering Visual, Acoustic and Glossary Features

Ananya Pandey, Dinesh Kumar Vishwakarma

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2408.07445 2024-08-15 cs.CV 83%

Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach

Muhammad Saad Saeed, Shah Nawaz, Muhammad Zaigham Zaheer, Muhammad Haris Khan, Karthik Nandakumar, Muhammad Haroon Yousaf, Hassan Sajjad, Tom De Schepper, Markus Schedl

专题命中音频语音多模态：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV

2408.01319 2024-08-05 cs.AI 83%

A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

Jiaqi Wang, Hanqi Jiang, Yiheng Liu, Chong Ma, Xu Zhang, Yi Pan, Mengyuan Liu, Peiran Gu, Sichen Xia, Wenjun Li, Yutong Zhang, Zihao Wu, Zhengliang Liu, Tianyang Zhong, Bao Ge, Tuo Zhang, Ning Qiang, Xintao Hu, Xi Jiang, Xin Zhang, Wei Zhang, Dinggang Shen, Tianming Liu, Shu Zhang

专题命中音频语音多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.AI

2407.17974 2024-07-26 cs.CL 83%

What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models

Tessa Verhoef, Kiana Shahrasbi, Tom Kouwenhoven

专题命中音频语音多模态：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CL

Comments Appeared at the 13th edition of the Workshop on Cognitive Modeling and Computational Linguistics (CMCL 2024)

2405.16640 2024-07-19 cs.AI cs.CL cs.CV cs.MM 83%

A Survey of Multimodal Large Language Model from A Data-centric Perspective

Tianyi Bai, Hao Liang, Binwang Wan, Yanran Xu, Xi Li, Shiyu Li, Ling Yang, Bozhou Li, Yifan Wang, Bin Cui, Ping Huang, Jiulong Shan, Conghui He, Binhang Yuan, Wentao Zhang

专题命中音频语音多模态：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

2405.10690 2024-07-16 cs.CV 83%

CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing

Faegheh Sardari, Armin Mustafa, Philip J. B. Jackson, Adrian Hilton

专题命中音频语音多模态：audio-visual(title,abstract)；cross-modal(abstract)；分类 cs.CV

Comments Accepted at ECCV 2024

2407.01034 2024-07-02 cs.CV cs.GR 83%

Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert

Han EunGi, Oh Hyun-Bin, Kim Sung-Bin, Corentin Nivelet Etcheberry, Suekyeong Nam, Janghoon Joo, Tae-Hyun Oh

专题命中音频语音多模态：audio-visual(title,abstract)；multimodal(abstract)；分类 cs.CV

Comments INTERSPEECH 2024

2406.17605 2024-06-26 cs.MM cs.AI cs.CL cs.CV cs.IR 83%

NativE: Multi-modal Knowledge Graph Completion in the Wild

Yichi Zhang, Zhuo Chen, Lingbing Guo, Yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, Huajun Chen

专题命中音频语音多模态：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by SIGIR 2024 as a full paper

2406.13894 2024-06-21 cs.CV cs.CY 83%

Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events

Mohammad Abu Tami, Huthaifa I. Ashqar, Mohammed Elhenawy

专题命中音频语音多模态：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV

1. 音频语音多模态 4549 篇

Multi-modal expressive personality recognition in data non-ideal audiovisual based on multi-scale feature enhancement and modal augment

Look, Listen, and Answer: Overcoming Biases for Audio-Visual Question Answering

Rethinking Audio-Visual Adversarial Vulnerability from Temporal and Modality Perspectives

video-SALMONN-o1: Reasoning-enhanced Audio-visual Large Language Model

Target Speaker Lipreading by Audio-Visual Self-Distillation Pretraining and Speaker Adaptation

Towards Multimodal Empathetic Response Generation: A Rich Text-Speech-Vision Avatar-based Benchmark

Driver Assistance System Based on Multimodal Data Hazard Detection

Reading to Listen at the Cocktail Party: Multi-Modal Speech Separation

A Multimodal Emotion Recognition System: Integrating Facial Expressions, Body Movement, Speech, and Spoken Language

Improving Lip-synchrony in Direct Audio-Visual Speech-to-Speech Translation

Depression Detection and Analysis using Large Language Models on Textual and Audio-Visual Modalities

A Survey of Recent Advances and Challenges in Deep Audio-Visual Correlation Learning

Enhancing Lie Detection Accuracy: A Comparative Study of Classic ML, CNN, and GCN Models using Audio-Visual Features

SaSR-Net: Source-Aware Semantic Representation Network for Enhancing Audio-Visual Question Answering

Towards Multi-Modal Mastery: A 4.5B Parameter Truly Multi-Modal Small Language Model

Language-Guided Joint Audio-Visual Editing via One-Shot Adaptation

The Curse of Multi-Modalities: Evaluating Hallucinations of Large Multimodal Models across Language, Visual, and Audio

CPM: Class-conditional Prompting Machine for Audio-visual Segmentation

MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts

A Novel Audio-Visual Information Fusion System for Mental Disorders Detection

Multi-Task Multi-Modal Self-Supervised Learning for Facial Expression Recognition

VyAnG-Net: A Novel Multi-Modal Sarcasm Recognition Model by Uncovering Visual, Acoustic and Glossary Features

Modality Invariant Multimodal Learning to Handle Missing Modalities: A Single-Branch Approach

A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

What does Kiki look like? Cross-modal associations between speech sounds and visual shapes in vision-and-language models

A Survey of Multimodal Large Language Model from A Data-centric Perspective

CoLeaF: A Contrastive-Collaborative Learning Framework for Weakly Supervised Audio-Visual Video Parsing

Enhancing Speech-Driven 3D Facial Animation with Audio-Visual Guidance from Lip Reading Expert

NativE: Multi-modal Knowledge Graph Completion in the Wild

Using Multimodal Large Language Models for Automated Detection of Traffic Safety Critical Events