多模态大模型

MAGIC: Map-Guided Few-Shot Audio-Visual Acoustics Modeling

Diwei Huang, Kunyang Lin, Peihao Chen, Qing Du, Mingkui Tan

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Comments 17 pages, 12 pages for main paper, 5 pages for supplementary

Audio-Visual Speech Representation Expert for Enhanced Talking Face Video Generation and Evaluation

Dogucan Yaman, Fevziye Irem Eyiokur, Leonard Bärmann, Seymanur Aktı, Hazım Kemal Ekenel, Alexander Waibel

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Comments CVPR2024 NTIRE Workshop

OmniActions: Predicting Digital Actions in Response to Real-World Multimodal Sensory Inputs with LLMs

Jiahao Nick Li, Yan Xu, Tovi Grossman, Stephanie Santosa, Michelle Li

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Comments Paper accepted to the 2024 CHI Conference on Human Factors in Computing Systems (CHI 2024)

Automatic Assessment of Dysarthria Using Audio-visual Vowel Graph Attention Network

Xiaokang Liu, Xiaoxia Du, Juan Liu, Rongfeng Su, Manwa Lawrence Ng, Yumei Zhang, Yudong Yang, Shaofeng Zhao, Lan Wang, Nan Yan

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Comments 10 pages, 7 figures, 7 tables

MMGER: Multi-modal and Multi-granularity Generative Error Correction with LLM for Joint Accent and Speech Recognition

Bingshen Mu, Yangze Li, Qijie Shao, Kun Wei, Xucheng Wan, Naijun Zheng, Huan Zhou, Lei Xie

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 eess.AS

ReactGenie: A Development Framework for Complex Multimodal Interactions Using Large Language Models

Jackie Junrui Yang, Yingtian Shi, Yuhan Zhang, Karina Li, Daniel Wan Rosli, Anisha Jain, Shuning Zhang, Tianshi Li, James A. Landay, Monica S. Lam

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

M3BAT: Unsupervised Domain Adaptation for Multimodal Mobile Sensing with Multi-Branch Adversarial Training

Lakmal Meegahapola, Hamza Hassoune, Daniel Gatica-Perez

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Comments Accepted at the Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies (IMWUT). Paper will be presented at ACM UbiComp 2024

AV-RIR: Audio-Visual Room Impulse Response Estimation

Anton Ratnarajah, Sreyan Ghosh, Sonal Kumar, Purva Chiniya, Dinesh Manocha

专题命中 音频语音多模态 ：audio-visual(title)；multi-modal(abstract)；分类 cs.CV

Comments Accepted to CVPR 2024

Fuse after Align: Improving Face-Voice Association Learning via Multimodal Encoder

Chong Peng, Liqiang He, Dan Su

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV

Multimodal Contextual Dialogue Breakdown Detection for Conversational AI Models

Md Messal Monem Miah, Ulie Schnaithmann, Arushi Raghuvanshi, Youngseo Son

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments Published in NAACL 2024 Industry Track

TIM: A Time Interval Machine for Audio-Visual Action Recognition

Jacob Chalk, Jaesung Huh, Evangelos Kazakos, Andrew Zisserman, Dima Damen

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Comments Accepted to CVPR 2024. Project Webpage: https://jacobchalk.github.io/TIM-Project

MMCert: Provable Defense against Adversarial Attacks to Multi-modal Models

Yanting Wang, Hongye Fu, Wei Zou, Jinyuan Jia

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CV

Comments To appear in CVPR'24

JMI at SemEval 2024 Task 3: Two-step approach for multimodal ECAC using in-context learning with GPT and instruction-tuned Llama models

Arefa, Mohammed Abbas Ansari, Chandni Saxena, Tanvir Ahmad

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Comments Paper Accepted at SemEval 2024

Audio-Visual Compound Expression Recognition Method based on Late Modality Fusion and Rule-based Decision

Elena Ryumina, Maxim Markitantov, Dmitry Ryumin, Heysem Kaya, Alexey Karpov

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Comments 7 pages, 3 figures

Cross-Attention is Not Always Needed: Dynamic Cross-Attention for Audio-Visual Dimensional Emotion Recognition

R. Gnana Praveen, Jahangir Alam

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Comments Accepted at IEEE ICME2024

How to Cache Important Contents for Multi-modal Service in Dynamic Networks: A DRL-based Caching Scheme

Zhe Zhang, Marc St-Hilaire, Xin Wei, Haiwei Dong, Abdulmotaleb El Saddik

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.MM

Journal ref IEEE Transactions on Multimedia (Early Access), 2024

ConvoFusion: Multi-Modal Conversational Diffusion for Co-Speech Gesture Synthesis

Muhammad Hamza Mughal, Rishabh Dabral, Ikhsanul Habibie, Lucia Donatelli, Marc Habermann, Christian Theobalt

专题命中 音频语音多模态 ：multi-modal(title,abstract)；分类 cs.CV

Comments CVPR 2024. Project Page: https://vcai.mpi-inf.mpg.de/projects/ConvoFusion/

An Audio-Visual Speech Separation Model Inspired by Cortico-Thalamo-Cortical Circuits

Kai Li, Fenghua Xie, Hang Chen, Kexin Yuan, Xiaolin Hu

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Comments Accepted by TPAMI 2024

Listen to Look into the Future: Audio-Visual Egocentric Gaze Anticipation

Bolin Lai, Fiona Ryan, Wenqi Jia, Miao Liu, James M. Rehg

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Comments 30 pages

Fine-Grained Engine Fault Sound Event Detection Using Multimodal Signals

Dennis Fedorishin, Livio Forte, Philip Schneider, Srirangaraj Setlur, Venu Govindaraju

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 eess.AS

Comments Accepted to ICASSP 2024

Latent Object Characteristics Recognition with Visual to Haptic-Audio Cross-modal Transfer Learning

Namiko Saito, Joao Moura, Hiroki Uchida, Sethu Vijayakumar

专题命中 音频语音多模态 ：cross-modal(title,abstract)；分类 cs.CV

Comments 8 pages

Amharic LLaMA and LLaVA: Multimodal LLMs for Low Resource Languages

Michael Andersland

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

A Modular End-to-End Multimodal Learning Method for Structured and Unstructured Data

Marco D Alessandro, Enrique Calabrés, Mikel Elkano

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.AI

Comments 8 pages, 1 figure

AVI-Talking: Learning Audio-Visual Instructions for Expressive 3D Talking Face Generation

Yasheng Sun, Wenqing Chu, Hang Zhou, Kaisiyuan Wang, Hideki Koike

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

Real-Time Idling Vehicles Detection using Combined Audio-Visual Deep Learning

Xiwen Li, Tristalee Mangin, Surojit Saha, Evan Blanchard, Dillon Tang, Henry Poppe, Nathan Searle, Ouk Choi, Kerry Kelly, Ross Whitaker

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 cs.CV

VL-Trojan: Multimodal Instruction Backdoor Attacks against Autoregressive Visual Language Models

Jiawei Liang, Siyuan Liang, Man Luo, Aishan Liu, Dongchen Han, Ee-Chien Chang, Xiaochun Cao

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CV

MMMModal -- Multi-Images Multi-Audio Multi-turn Multi-Modal

Husein Zolkepli, Aisyah Razak, Kamarul Adha, Ariff Nazhan

专题命中 音频语音多模态 ：multi-modal(title)；multimodal(abstract)；分类 cs.CL

MasonPerplexity at Multimodal Hate Speech Event Detection 2024: Hate Speech and Target Detection Using Transformer Ensembles

Amrita Ganguly, Al Nahian Bin Emran, Sadiya Sayara Chowdhury Puspo, Md Nishat Raihan, Dhiman Goswami, Marcos Zampieri

专题命中 音频语音多模态 ：multimodal(title,abstract)；分类 cs.CL

Overview of the L3DAS23 Challenge on Audio-Visual Extended Reality

Christian Marinoni, Riccardo Fosco Gramaccioni, Changan Chen, Aurelio Uncini, Danilo Comminiello

专题命中 音频语音多模态 ：audio-visual(title,abstract)；分类 eess.AS

Comments Accepted to 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP 2023)

Multi-Modal Emotion Recognition by Text, Speech and Video Using Pretrained Transformers

Minoo Shayaninasab, Bagher Babaali

专题命中 音频语音多模态 ：multi-modal(title)；multimodal(abstract)；分类 cs.AI