多模态大模型

FreeVA: Offline MLLM as Training-Free Video Assistant

Wenhao Wu

专题命中 视频多模态 ：MLLM(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments Preprint. Work in progress

SpikeMba: Multi-Modal Spiking Saliency Mamba for Temporal Video Grounding

Wenrui Li, Xiaopeng Hong, Ruiqin Xiong, Xiaopeng Fan

专题命中 视频多模态 ：multi-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

VideoAdviser: Video Knowledge Distillation for Multimodal Transfer Learning

Yanan Wang, Donghuo Zeng, Shinya Wada, Satoshi Kurihara

专题命中 视频多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL

Comments Accepted by IEEE Access

Journal ref in IEEE Access, vol. 11, pp. 51229-51240, 2023

Active Acquisition for Multimodal Temporal Data: A Challenging Decision-Making Task

Jannik Kossen, Cătălina Cangea, Eszter Vértes, Andrew Jaegle, Viorica Patraucean, Ira Ktena, Nenad Tomasev, Danielle Belgrave

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Published in Transactions on Machine Learning Research. Previous version accepted to Foundation Models for Decision Making Workshop at NeurIPS 2022

A Unified Multimodal De- and Re-coupling Framework for RGB-D Motion Recognition

Benjia Zhou, Pichao Wang, Jun Wan, Yanyan Liang, Fan Wang

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted to TPAMI 2023

MMG-Ego4D: Multi-Modal Generalization in Egocentric Action Recognition

Xinyu Gong, Sreyas Mohan, Naina Dhingra, Jean-Charles Bazin, Yilei Li, Zhangyang Wang, Rakesh Ranjan

专题命中 视频多模态 ：multi-modal(title)；multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted to CVPR 2023

Scalable and Accurate Self-supervised Multimodal Representation Learning without Aligned Video and Text Data

Vladislav Lialin, Stephen Rawls, David Chan, Shalini Ghosh, Anna Rumshisky, Wael Hamza

专题命中 视频多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Journal ref 2023 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW)

Frame-wise Cross-modal Matching for Video Moment Retrieval

Haoyu Tang, Jihua Zhu, Meng Liu, Zan Gao, Zhiyong Cheng

专题命中 视频多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Comments 12 pages; accepted by IEEE TMM

Journal ref IEEE Transactions on Multimedia 2021

Multimodal Event Transformer for Image-guided Story Ending Generation

Yucheng Zhou, Guodong Long

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments EACL 2023

Rethinking Multi-Modal Alignment in Video Question Answering from Feature and Sample Perspectives

Shaoning Xiao, Long Chen, Kaifeng Gao, Zhao Wang, Yi Yang, Zhimeng Zhang, Jun Xiao

专题命中 视频多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Progressive Cross-modal Knowledge Distillation for Human Action Recognition

Jianyuan Ni, Anne H. H. Ngu, Yan Yan

专题命中 视频多模态 ：cross-modal(title)；multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.MM

Comments ACM MM 2022

On Pursuit of Designing Multi-modal Transformer for Video Grounding

Meng Cao, Long Chen, Mike Zheng Shou, Can Zhang, Yuexian Zou

专题命中 视频多模态 ：multi-modal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted by Conference on Empirical Methods in Natural Language Processing (EMNLP 2021, Oral)

Modeling Temporal-Modal Entity Graph for Procedural Multimodal Machine Comprehension

Huibin Zhang, Zhengkun Zhang, Yao Zhang, Jun Wang, Yufan Li, Ning jiang, Xin wei, Zhenglu Yang

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CL、cs.MM

Comments Accepted by ACL-2022

Cross-Modal Progressive Comprehension for Referring Segmentation

Si Liu, Tianrui Hui, Shaofei Huang, Yunchao Wei, Bo Li, Guanbin Li

专题命中 视频多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Comments Accepted by TPAMI 2021

Dynamic Graph Representation Learning for Video Dialog via Multi-Modal Shuffled Transformers

Shijie Geng, Peng Gao, Moitreya Chatterjee, Chiori Hori, Jonathan Le Roux, Yongfeng Zhang, Hongsheng Li, Anoop Cherian

专题命中 视频多模态 ：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL

Comments Accepted at AAAI 2021

Video Question Generation via Cross-Modal Self-Attention Networks Learning

Yu-Siang Wang, Hung-Ting Su, Chen-Hsi Chang, Zhe-Yu Liu, Winston H. Hsu

专题命中 视频多模态 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted by ICASSP 2020

Learning Shared Semantic Space with Correlation Alignment for Cross-modal Event Retrieval

Zhenguo Yang, Zehang Lin, Peipei Kang, Jianming Lv, Qing Li, Wenyin Liu

专题命中 视频多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM

Comments 22 pages, submitted to ACM Transactions on Multimedia Computing Communications and Applications(ACM TOMM)

From FiLM to Video: Multi-turn Question Answering with Multi-modal Context

Dat Tien Nguyen, Shikhar Sharma, Hannes Schulz, Layla El Asri

专题命中 视频多模态 ：multi-modal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.CL

Comments Accepted for an Oral presentation at the DSTC7 workshop at AAAI 2019

Zero-Shot Event Detection by Multimodal Distributional Semantic Embedding of Videos

Mohamed Elhoseiny, Jingen Liu, Hui Cheng, Harpreet Sawhney, Ahmed Elgammal

专题命中 视频多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments To appear in AAAI 2016

Multimodal Approach for Video Surveillance Indexing and Retrieval

Ali Wali, Adel M. Alimi

专题命中 视频多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.CV、cs.MM

Comments 7 pages

Journal ref Journal of Intelligent Computing, Volume: 1, Issue: 4 (December 2010), Page: 165-175

Language as the Medium: Multimodal Video Classification through text only

Laura Hanu, Anita L. Verő, James Thewlis

专题命中 视频多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL、cs.AI；multimodal foundation model(comments)

Comments Accepted at "What is Next in Multimodal Foundation Models?" (MMFM) workshop at ICCV 2023

Med-CRAFT: An Information System for Explainable and Configurable Construction of Multimodal Medical QA Datasets

Med-CRAFT：通过知识图谱遍历自动构建可解释的多跳视频工作负载

Shenxi Liu, Kan Li, Mingyang Zhao, Yuhang Tian, Bin Li

机构 * Beijing Institute of Technology（北京理工大学） ； The Hong Kong Polytechnic University（香港理工大学） ； Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences（中国科学院深圳先进技术研究所） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；MLLM(abstract_cn)；分类 cs.AI

AI总结 Med-CRAFT通过知识图谱遍历自动构建可解释的多跳视频工作负载，生成具有细粒度时间选择性和多跳逻辑复杂性的医疗视频推理基准。

Comments 23 pages, 4 figures, 10 tables

Implicit Neural Representations for Multimodal Longitudinal Image Imputation and Interpolation

用于多模态纵向图像插补与插值的隐式神经表示

Sina Wendrich, Lukas Förner, Zoe Reinke, Kartikay Tehlan, Ansgar Berlis, Michael Frühwald, Matthias Wagner, Thomas Wendler

机构 * University Hospital Augsburg（奥格斯堡大学医院） ； University of Augsburg（奥格斯堡大学） ； Technical University of Munich（慕尼黑工业大学） ； Bavarian Cancer Research Center (BZKF)（巴伐利亚癌症研究中心） ； Swabian Children’s Cancer Center（施瓦本儿童癌症中心） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

AI总结 该研究针对临床纵向MRI数据缺失等问题，提出条件隐式神经表示模型，经儿科脑肿瘤数据验证，可显著改进插值效果，置信度估计可靠。

Farm-LightSeek: An Edge-centric Multimodal Agricultural IoT Data Analytics Framework with Lightweight LLMs

Farm-LightSeek：一种以边缘为中心的多模态农业物联网数据分析框架，集成轻量级语言模型

Dawen Jiang, Zhishu Shen, Qiushi Zheng, Tiehua Zhang, Wei Xiang, Jiong Jin

机构 * School of Computer Science and Artificial Intelligence, Wuhan University of Technology（武汉理工大学计算机科学与人工智能学院） ； School of Engineering, Swinburne University of Technology（斯威本科技大学工程学院） ； School of Computer Science and Technology, Tongji University（同济大学计算机科学与技术学院） ； School of Computing, Engineering and Mathematical Sciences, La Trobe University（拉筹伯大学计算、工程与数学科学学院） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

AI总结 针对智能农业面临的挑战，提出Farm-LightSeek框架，将大语言模型与边缘计算集成。通过传感器收集多源数据，在边缘节点进行跨模态推理等。创新包括闭环架构等，实验表明该框架在关键任务中性能可靠，推动了智能实时农业及两者深度集成。

Comments Accepted by IEEE Internet of Things Magazine

Beyond Independent Optimization: Compression, MoE Routing, and Quantization Interactions in Multimodal Edge Intelligence

超越独立优化：多模态边缘智能中的压缩、混合专家路由和量化交互

Jay Gor, Karm Dave, Akshita Abrol, Rajesh Gupta, Sudeep Tanwar, Zhengkui Wang

机构 * Nirma University（尼玛大学） ； Singapore Institute of Technology（新加坡理工学院） ； Marwadi University（马尔瓦迪大学） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.AI

AI总结 研究多模态边缘智能中高效推理受多种因素限制，回顾相关模型进展，指出技术间相互影响不能独立优化，介绍关键设计权衡，引入视频MoE模型诊断方法，强调多方面开放研究方向。

Learning to Detect Cross-Modal Negation: An Analysis of Latent Representations and an Attention-Based Solution

学习检测跨模态否定：潜在表示分析与基于注意力的解决方案

Ali AbuSaleh, Leon Hammerla, Alexander Mehler

专题命中 视频多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.AI

AI总结 研究跨模态否定检测难题，提出新型跨模态注意力架构，分析发现文本与视觉否定的不对称性，结合自监督视频表示推进时间否定建模，为多模态系统学习语义对齐表示提供新方法。

Comments This manuscript is an accepted version of the article (published at ICNLP2026). Published in IEEE Xplore, DOI:10.1109/ICNLP69856.2026.11527861 document: https://ieeexplore.ieee.org/abstract/document/11527861

Journal ref 2026 8th International Conference on Natural Language Processing (ICNLP), Xi'an, China, 2026, pp. 613-622

Multimodal Video-to-Music Recommendation via Semantic Retrieval and Temporal Reranking

通过语义检索和时间重排实现多模态视频到音乐推荐

Seungheon Doh, Minhee Lee, Sangmoon Lee, Ben Sangbae Chon, Juhan Nam

机构 * Graduate School of Culture Technology, KAIST（韩国釜山科学技术大学文化科技研究生院） ； Gaudio Lab, Inc（Gaudio实验室） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；audio-visual(abstract)；分类 cs.MM

AI总结 提出用于视频到音乐推荐的两阶段框架VTMR，第一阶段通过全局嵌入检索语义兼容候选，第二阶段关注时间序列重排。实验显示该框架能提升推荐效果，人类偏好研究表明其在总体偏好上与商业基线相当，音乐质量优于生成基线。

Comments Accepted for publication at The Machine Learning for Audio workshop at ICML 2026

Multimodal Ambivalence/Hesitancy Recognition in Videos for Personalized Digital Health Interventions

视频中矛盾/犹豫识别用于个性化数字健康干预

Manuela González-González, Soufiane Belharbi, Muhammad Osama Zeeshan, Masoumeh Sharafi, Muhammad Haseeb Aslam, Lorenzo Sia, Nicolas Richet, Marco Pedersoli, Alessandro Lameiras Koerich, Simon L Bacon, Eric Granger

机构 * LIVIA, Dept. of Systems Engineering, ETS Montreal, Canada（ETS蒙特利尔大学系统工程系LIVIA实验室） ； LIVIA, Dept. of Software and IT Engineering, ETS Montreal, Canada（ETS蒙特利尔大学软件与信息工程系LIVIA实验室） ； Dept. of Health, Kinesiology, & Applied Physiology, Concordia University, Montreal, Canada（康科迪亚大学健康、运动科学与应用生理学系） ； Montreal Behavioural Medicine Centre, CIUSSS Nord-de-l’Ile-de-Montréal, Canada（蒙特利尔行为医学中心，蒙特利尔北岛卫生与社会服务局） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；multi-modal(abstract)；分类 cs.CV

AI总结 本文研究了通过深度学习模型在视频中进行矛盾/犹豫识别，以提升数字健康干预的个性化和成本效益，实验基于新的BAH视频数据集，发现需改进多模态模型以准确识别矛盾/犹豫。

Comments 11 pages, 4 figures, ACII 2026. arXiv admin note: substantial text overlap with arXiv:2505.19328

Temporal and Cross-Modal Alignment for Enhanced Audiovisual Video Captioning

增强视听视频字幕的时间与跨模态对齐

Chen Zhao, Jiajun Ma, Qilong Huang, Tiehan Fan, Hongyu Li, Zhuoliang Kang, Xiaoming Wei, Jian Yang, Ying Tai

机构 * Nanjing University, State Key Laboratory for Novel Software Technology（南京大学，计算机软件新技术国家重点实验室） ； Meituan（美团） 纠错

专题命中 视频多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV

AI总结 提出TCA-Captioner框架，通过观察-检查-纠正迭代策略和密集交互数据集，解决视听字幕中的模态分离与时间不一致问题，实现精准的视听绑定与因果动态建模。

Comments ECCV 2026

MLCR: Multi-Level Cue Refinement for Long-Term Multimodal Action Quality Assessment

PIDNet: 逐步隐式解耦网络用于多模态动作质量评估

Qiqi Li, Pengfei Wang, Hongyu Chen, Nenggan Zheng

机构 * Qiushi Academy for Advanced Studies (QAAS), Zhejiang University（浙江大学启斯特先进研究院） ； College of Computer Science and Technology, Zhejiang University（浙江大学计算机科学与技术学院） ； School of Software Technology, Zhejiang University（浙江大学软件学院） ； State Key Lab of Brain-Machine Intelligence（脑机智能国家重点实验室） ； Collaborative Innovation Center for Artificial Intelligence by MOE and Zhejiang Provincial Government (ZJU)（教育部-浙江省人工智能协同创新中心） ； Zhejiang Lab（浙江实验室） 纠错

专题命中 视频多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV

AI总结 本文提出PIDNet，通过逐步整合多模态信息与全局质量语义，提升多模态动作质量评估的准确性。采用iMambaWave模块和三阶段融合网络，有效解耦模态特定信息并增强特征表示。

Comments 14 pages, 6 figures, 11 tables