多模态大模型

Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Tasks

Gabriela Sejnova, Michal Vavrecka, Karla Stepanova

机构 * Czech Institute of Informatics, Robotics and Cybernetics（捷克信息学、机器人学与自动控制研究所） ； Czech Technical University in Prague（布拉格捷克技术大学） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；image-text(abstract)

Comments 7 pages, 5 figures, 2 tables, conference

Journal ref 2024 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)

Spiking Neural Networks with Temporal Attention-Guided Adaptive Fusion for imbalanced Multi-modal Learning

Jiangrong Shen, Yulin Xie, Qi Xu, Gang Pan, Huajin Tang, Badong Chen

机构 * Faculty of Electronic and Information Engineering, Xi’an Jiaotong University（电子与信息工程学院，西安交通大学） ； Institute of Artificial Intelligence and Robotics, Xi’an Jiaotong University（人工智能与机器人研究院，西安交通大学） ； State Key Lab of Brain-Machine Intelligence, Zhejiang University（脑机智能国家重点实验室，浙江大学） ； School of Computer Science, Dalian University of Technology（计算机科学学院，大连理工大学） ； National Key Lab of Human-Machine Hybrid Augmented Intelligence, Xi’an Jiaotong University（人机混合增强智能国家实验室，西安交通大学） 纠错

专题命中 视频多模态 ：multi-modal(title)；multimodal(abstract)；cross-modal(abstract)

Enhanced Multimodal Hate Video Detection via Channel-wise and Modality-wise Fusion

Yinghui Zhang, Tailin Chen, Yuchen Zhang, Zeyu Fu

机构 * Department of Computer Science, University of Exeter（埃克塞特大学计算机科学系） ； Institute for Analytics and Data Science, University of Essex（埃塞克斯大学分析与数据科学研究所） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments ICDMW 2024, Github: https://github.com/EvelynZ10/cmfusion

Journal ref 2024 IEEE International Conference on Data Mining Workshops (ICDMW), Abu Dhabi, United Arab Emirates, 2024, pp. 183-190

Hybrid-Level Instruction Injection for Video Token Compression in Multi-modal Large Language Models

Zhihang Liu, Chen-Wei Xie, Pandeng Li, Liming Zhao, Longxiang Tang, Yun Zheng, Chuanbin Liu, Hongtao Xie

专题命中 视频多模态 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to CVPR2025

CREMA: Generalizable and Efficient Video-Language Reasoning via Multimodal Modular Fusion

Shoubin Yu, Jaehong Yoon, Mohit Bansal

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICLR 2025; first two authors contributed equally. Project page: https://CREMA-VideoLLM.github.io/

Lost in Time: Clock and Calendar Understanding Challenges in Multimodal LLMs

Rohit Saxena, Aryo Pradipta Gema, Pasquale Minervini

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at the ICLR 2025 Workshop on Reasoning and Planning for Large Language Models

Watch Out E-scooter Coming Through: Multimodal Sensing of Mixed Traffic Use and Conflicts Through Riders Ego-centric Views

Hiruni Nuwanthika Kegalle, Danula Hettiachchi, Jeffrey Chan, Mark Sanderson, Flora D. Salim

专题命中 视频多模态 ：multimodal(title,abstract)；multi-modal(abstract)

Comments Accepted in Proc. ACM Interactive, Mobile, Wearable and Ubiquitous Technologies,(March 2025), 23 pages. https://doi.org/10.1145/3712284

Modality-Invariant Bidirectional Temporal Representation Distillation Network for Missing Multimodal Sentiment Analysis

Xincheng Wang, Liejun Wang, Yinfeng Yu, Xinxin Jiao

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、eess.AS

Comments Accepted for publication by 2025 IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2025)

Towards Expressive Video Dubbing with Multiscale Multimodal Context Interaction

Yuan Zhao, Rui Liu, Gaoxiang Cong

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted by ICASSP 2025

Enhancing Multimodal Affective Analysis with Learned Live Comment Features

Zhaoyuan Deng, Amith Ananthram, Kathleen McKeown

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI、cs.MM

TemporalBench: Benchmarking Fine-grained Temporal Understanding for Multimodal Video Models

Mu Cai, Reuben Tan, Jianrui Zhang, Bocheng Zou, Kai Zhang, Feng Yao, Fangrui Zhu, Jing Gu, Yiwu Zhong, Yuzhang Shang, Yao Dou, Jaden Park, Jianfeng Gao, Yong Jae Lee, Jianwei Yang

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://temporalbench.github.io/

TV-TREES: Multimodal Entailment Trees for Neuro-Symbolic Video Reasoning

Kate Sanders, Nathaniel Weir, Benjamin Van Durme

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 9 pages, EMNLP 2024

Enhancing Adverse Drug Event Detection with Multimodal Dataset: Corpus Creation and Model Development

Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Aman Chadha, Samrat Mondal

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL Findings 2024

MCDubber: Multimodal Context-Aware Expressive Video Dubbing

Yuan Zhao, Zhenqi Jia, Rui Liu, De Hu, Feilong Bao, Guanglai Gao

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.MM、eess.AS

Comments Accepted by NCMMSC2024

VHAKG: A Multi-modal Knowledge Graph Based on Synchronized Multi-view Videos of Daily Activities

Shusaku Egami, Takahiro Ugai, Swe Nwe Nwe Htun, Ken Fukuda

专题命中 视频多模态 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 5 pages, 4 figures, accepted by CIKM2024 Resource Track

End-to-end Semantic-centric Video-based Multimodal Affective Computing

Ronghao Lin, Ying Zeng, Sijie Mai, Haifeng Hu

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Under Review

Weakly Supervised Gaussian Contrastive Grounding with Large Multimodal Models for Video Question Answering

Haibo Wang, Chenghang Lai, Yixuan Sun, Weifeng Ge

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments accepted by ACM Multimedia 2024

MMToM-QA: Multimodal Theory of Mind Question Answering

Chuanyang Jin, Yutong Wu, Jing Cao, Jiannan Xiang, Yen-Ling Kuo, Zhiting Hu, Tomer Ullman, Antonio Torralba, Joshua B. Tenenbaum, Tianmin Shu

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2024. 26 pages, 11 figures, 7 tables

Missingness-resilient Video-enhanced Multimodal Disfluency Detection

Payal Mohapatra, Shamika Likhite, Subrata Biswas, Bashima Islam, Qi Zhu

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.MM、eess.AS

Comments Accepted to Interspeech 2024

Towards Multi-Task Multi-Modal Models: A Video Generative Perspective

Lijun Yu

专题命中 视频多模态 ：multi-modal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments PhD thesis

GlitchBench: Can large multimodal models detect video game glitches?

Mohammad Reza Taesiri, Tianjun Feng, Anh Nguyen, Cor-Paul Bezemer

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2024

TimeChat: A Time-sensitive Multimodal Large Language Model for Long Video Understanding

Shuhuai Ren, Linli Yao, Shicheng Li, Xu Sun, Lu Hou

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2024 camera-ready version, code is available at https://github.com/RenShuhuai-Andy/TimeChat

Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action

Jiasen Lu, Christopher Clark, Sangho Lee, Zichen Zhang, Savya Khosla, Ryan Marten, Derek Hoiem, Aniruddha Kembhavi

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 38 pages, 20 figures

RTQ: Rethinking Video-language Understanding Based on Image-text Model

Xiao Wang, Yaoyu Li, Tian Gan, Zheng Zhang, Jingjing Lv, Liqiang Nie

专题命中 视频多模态 ：image-text(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted by ACM MM 2023 as Oral representation

Journal ref In International Conference on Multimedia. ACM, 557--566 (2023)

MAiVAR-T: Multimodal Audio-image and Video Action Recognizer using Transformers

Muhammad Bilal Shaikh, Douglas Chai, Syed Mohammed Shamsul Islam, Naveed Akhtar

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 6 pages, 7 figures, 4 tables, Peer reviewed, Accepted @ The 11th European Workshop on Visual Information Processing (EUVIP) will be held on 11th-14th September 2023, in Gjøvik, Norway. arXiv admin note: text overlap with arXiv:2103.15691 by other authors

C2KD: Cross-Lingual Cross-Modal Knowledge Distillation for Multilingual Text-Video Retrieval

Andrew Rouditchenko, Yung-Sung Chuang, Nina Shvetsova, Samuel Thomas, Rogerio Feris, Brian Kingsbury, Leonid Karlinsky, David Harwath, Hilde Kuehne, James Glass

专题命中 视频多模态 ：cross-modal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted at ICASSP 2023. The code, models, and dataset are available at https://github.com/roudimit/c2kd

HateMM: A Multi-Modal Dataset for Hate Video Classification

Mithun Das, Rohit Raj, Punyajoy Saha, Binny Mathew, Manish Gupta, Animesh Mukherjee

专题命中 视频多模态 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted at ICWSM 2023(dataset track)

Multi-modal imaging using a cascaded microscope design

Xi Yang, Mark Harfouche, Kevin C. Zhou, Lucas Kreiss, Shiqi Xu, Kanghyun Kim, Roarke Horstmeyer

专题命中 视频多模态 ：multi-modal(title,abstract)；multimodal(abstract)

Multimodal Frame-Scoring Transformer for Video Summarization

Jeiyoon Park, Kiho Kwoun, Chanhee Lee, Heuiseok Lim

专题命中 视频多模态 ：multimodal(title,abstract)；audio-visual(abstract)

Comments preprint

TLDW: Extreme Multimodal Summarisation of News Videos

Peggy Tang, Kun Hu, Lei Zhang, Jiebo Luo, Zhiyong Wang

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.MM