多模态大模型

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, Hui Wang

专题命中 视频多模态 ：multi-modal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 5 figures

Video-LaVIT: Unified Video-Language Pre-training with Decoupled Visual-Motional Tokenization

Yang Jin, Zhicheng Sun, Kun Xu, Kun Xu, Liwei Chen, Hao Jiang, Quzhe Huang, Chengru Song, Yuliang Liu, Di Zhang, Yang Song, Kun Gai, Yadong Mu

专题命中 视频多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Artemis: Towards Referential Understanding in Complex Videos

Jihao Qiu, Yuan Zhang, Xi Tang, Lingxi Xie, Tianren Ma, Pengyu Yan, David Doermann, Qixiang Ye, Yunjie Tian

专题命中 视频多模态 ：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 19 pages, 14 figures. Code and data are available at https://github.com/qiujihao19/Artemis

Abductive Ego-View Accident Video Understanding for Safe Driving Perception

Jianwu Fang, Lei-lei Li, Junfei Zhou, Junbin Xiao, Hongkai Yu, Chen Lv, Jianru Xue, Tat-Seng Chua

专题命中 视频多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR2024. This is not the camera-ready version. The Project page: http://www.lotvsmmau.net

SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Tasks

Xingning Dong, Qingpei Guo, Tian Gan, Qing Wang, Jianlong Wu, Xiangyuan Ren, Yuan Cheng, Wei Chu

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by TCSVT (IEEE Transactions on Circuits and Systems for Video Technology)

Conversation Understanding using Relational Temporal Graph Neural Networks with Auxiliary Cross-Modality Interaction

Cam-Van Thi Nguyen, Anh-Tuan Mai, The-Son Le, Hai-Dang Kieu, Duc-Trong Le

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CL、cs.MM

Comments EMNLP 2023

Journal ref The 2023 Conference on Empirical Methods in Natural Language Processing

Learning Unseen Modality Interaction

Yunhua Zhang, Hazel Doughty, Cees G. M. Snoek

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Published at NeurIPS 2023

Improving Vision Anomaly Detection with the Guidance of Language Modality

Dong Chen, Kaihang Pan, Guoming Wang, Yueting Zhuang, Siliang Tang

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 10 figures

ViGT: Proposal-free Video Grounding with Learnable Token in Transformer

Kun Li, Dan Guo, Meng Wang

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments This paper has been accepted by SCIENCE CHINA Information Sciences

$C^3$: Compositional Counterfactual Contrastive Learning for Video-grounded Dialogues

Hung Le, Nancy F. Chen, Steven C. H. Hoi

专题命中 视频多模态 ：multimodal(abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL

Comments 24th Meeting of the Special Interest Group on Discourse and Dialogue (SIGDIAL)

Self-Supervised Learning for Videos: A Survey

Madeline C. Schiappa, Yogesh S. Rawat, Mubarak Shah

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments ACM CSUR (December 2022). Project Link: https://bit.ly/3Oimc7Q

Journal ref ACM Comput. Surv. (December 2022)

Contrastive Video Question Answering via Video Graph Transformer

Junbin Xiao, Pan Zhou, Angela Yao, Yicong Li, Richang Hong, Shuicheng Yan, Tat-Seng Chua

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by IEEE T-PAMI'23

Youku-mPLUG: A 10 Million Large-scale Chinese Video-Language Dataset for Pre-training and Benchmarks

Haiyang Xu, Qinghao Ye, Xuan Wu, Ming Yan, Yuan Miao, Jiabo Ye, Guohai Xu, Anwen Hu, Yaya Shi, Guangwei Xu, Chenliang Li, Qi Qian, Maofei Que, Ji Zhang, Xiao Zeng, Fei Huang

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Working in progress

Faster Video Moment Retrieval with Point-Level Supervision

Xun Jiang, Zailei Zhou, Xing Xu, Yang Yang, Guoqing Wang, Heng Tao Shen

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments 10 pages, 7 figures

HIINT: Historical, Intra- and Inter- personal Dynamics Modeling with Cross-person Memory Transformer

Yubin Kim, Dong Won Lee, Paul Pu Liang, Sharifa Algohwinem, Cynthia Breazeal, Hae Won Park

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Parallel Attention Network with Sequence Matching for Video Grounding

Hao Zhang, Aixin Sun, Wei Jing, Liangli Zhen, Joey Tianyi Zhou, Rick Siow Mong Goh

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments 15 pages, 10 figures, 7 tables, Findings at ACL 2021

Generation-Guided Multi-Level Unified Network for Video Grounding

Xing Cheng, Xiangyu Wu, Dong Shen, Hezheng Lin, Fan Yang

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Revisiting Temporal Modeling for CLIP-based Image-to-Video Knowledge Transferring

Ruyang Liu, Jingjia Huang, Ge Li, Jiashi Feng, Xinglong Wu, Thomas H. Li

专题命中 视频多模态 ：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

CLOP: Video-and-Language Pre-Training with Knowledge Regularizations

Guohao Li, Hu Yang, Feng He, Zhifan Feng, Yajuan Lyu, Hua Wu, Haifeng Wang

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments ACM Multimedia 2022 (MM'22)

A Dataset for Medical Instructional Video Classification and Question Answering

Deepak Gupta, Kush Attal, Dina Demner-Fushman

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

VLM: Task-agnostic Video-Language Model Pre-training for Video Understanding

Hu Xu, Gargi Ghosh, Po-Yao Huang, Prahal Arora, Masoumeh Aminzadeh, Christoph Feichtenhofer, Florian Metze, Luke Zettlemoyer

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments 9 pages, ACL Findings 2021

VLG-Net: Video-Language Graph Matching Network for Video Grounding

Mattia Soldan, Mengmeng Xu, Sisi Qu, Jesper Tegner, Bernard Ghanem

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments 14 pages, 7 figures, In proceeding of the ICCV21 workshop: AI for Creative Video Editing and Understanding 2021

Collaborative Spatial-Temporal Modeling for Language-Queried Video Actor Segmentation

Tianrui Hui, Shaofei Huang, Si Liu, Zihan Ding, Guanbin Li, Wenguan Wang, Jizhong Han, Fei Wang

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by CVPR 2021

Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language

Hassan Akbari, Hamid Palangi, Jianwei Yang, Sudha Rao, Asli Celikyilmaz, Roland Fernandez, Paul Smolensky, Jianfeng Gao, Shih-Fu Chang

专题命中 视频多模态 ：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrieval

Jie Lei, Licheng Yu, Tamara L. Berg, Mohit Bansal

专题命中 视频多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments ECCV 2020 (extended version, with TVC dataset+models; 35 pages)

Dense-Caption Matching and Frame-Selection Gating for Temporal Localization in VideoQA

Hyounghun Kim, Zineng Tang, Mohit Bansal

专题命中 视频多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments ACL 2020 (11 pages)

Mapping Multimodal Pilot Stress and Fatigue During Flight Sessions

飞行训练期间多模态飞行员压力与疲劳的映射

Atandrila Chowdhury, Sudip Vhaduri, Julius Keller, Debra Henneberry, Mark Wilson

专题命中 视频多模态 ：multimodal(title)

AI总结 本研究结合生理信号与主观报告，分析学员飞行员飞行训练中的压力与疲劳模式，证实数据驱动方法可用于监测飞行员健康状态。

Comments Under Review

Multimodal Spatiotemporal Atmospheric Data Assimilation with Latent Video Flow-matching

基于潜在流匹配的多模态时空大气数据同化

Dibyajyoti Chakraborty, Romit Maulik

机构 * The Pennsylvania State University（宾夕法尼亚州立大学） ； Purdue University（普渡大学） 纠错

专题命中 视频多模态 ：multimodal(title)

AI总结 该研究提出基于潜在流匹配的多模态时空大气数据同化方法，利用ERA5再分析数据训练先验，结合后验采样同化真实观测，可完成多种DA任务，从稀疏观测生成的集合预报性能达先进水平。

Mitigating Detector Ageing Effects with Graph-Based Multi-Modal Track Reconstruction at Belle II

在 Belle II 中使用基于图的多模态轨道重建减轻探测器老化效应

Lea Reuter, Tristan Brandes, Giacomo De Pietro, Torben Ferber

专题命中 视频多模态 ：multi-modal(title)

AI总结 研究 Belle II 探测器老化对轨道寻找的影响，采用基于图神经网络的算法，将轨道寻找设为全局关系聚类问题，经全探测器模拟评估性能，结果显示该算法在探测器老化时能提高鲁棒性，稳定跟踪性能。

Comments proceedings for Connecting The Dots 2025

Multimodal Empirical Bayes Variational Autoencoders for Joint Longitudinal and Time-to-Event Modeling

用于联合纵向和事件发生时间建模的多模态经验贝叶斯变分自编码器

Anders Sjöberg, Nils Olsson, Marcus Baaz, Mats Jirstrand

机构 * Fraunhofer-Chalmers Centre（弗劳恩霍夫-查尔默斯中心） ； Department of Electrical Engineering（电气工程系） ； Chalmers University of Technology（查尔姆斯理工大学） 纠错

专题命中 视频多模态 ：multimodal(title)

AI总结 研究针对纵向肿瘤测量等多源数据整合难的问题，扩展EB-VAE框架用于联合纵向和事件发生时间建模，比较不同解码器公式，纳入基因组协变量，实现肿瘤生长等联合预测，确定相关基因指标，提供了灵活概率框架。