arXivDaily arXiv每日学术速递 周一至周五更新

AI 大模型

多模态大模型

跨文本、图像、视频、音频等模态的大模型与学习方法。

共收录 4726 信号源:cs.CV, cs.CL, cs.AI, cs.MM, eess.AS

1. 视频多模态 4726 篇

2512.06673 2026-05-12 cs.CV 77%

Detector-Empowered Video Large Language Model for Efficient Spatio-Temporal Grounding

基于检测器的视频大语言模型用于高效的时空定位

Shida Gao, Feng Xue, Xiangfeng Wang, Anlong Ming, Zhaowen Lin, Haiyang Zhang, Teng Long, Nicu Sebe, Yihua Shao, Haozhe Wang, Wei Wang

机构 * Beijing University of Posts and Telecommunications(北京邮电大学) University of Trento(特伦特大学) Institute of Automation, Chinese Academy of Sciences(中国科学院自动化研究所) Hong Kong University of Science and Technology(香港科技大学) ZTE Corporation(中兴通讯)

专题命中 视频多模态 :MLLM(abstract,abstract_cn);multimodal(abstract);分类 cs.CV

AI总结 本文提出DEViL模型,通过将密集空间定位任务转移给训练良好的检测器,提升视频时空定位的效率与性能,实现43.1%的m_vIoU和14.33 FPS的高效表现。

详情

展开后加载摘要…

URL PDF HTML 收藏
2605.01506 2026-05-05 cs.CV 77%

OmniEncoder: See, Hear, and Feel Continuous Motion Like Humans With One Encoder

OmniEncoder: 通过一个编码器实现如同人类般连续的视觉、听觉与触觉感知

Detao Bai, Shimin Yao, Weixuan Chen, Chengen Lai, Yuanming Li, Zhiheng Ma, Xihan Wei

机构 * Tongyi Lab Alibaba Group(阿里巴巴集团通义实验室) Shenzhen University of Advanced Technology(深圳先进技术大学)

专题命中 视频多模态 :cross-modal(abstract);audio-visual(abstract);omni-modal(abstract);分类 cs.CV

AI总结 OmniEncoder通过统一的Transformer架构,在共享潜在空间中对视觉和音频信号以25fps对称嵌入,解决多模态解耦与计算效率问题,提升连续视觉理解任务性能。

详情

展开后加载摘要…

URL PDF HTML 收藏
2604.08014 2026-04-22 cs.CV 77%

Bridging Time and Space: Decoupled Spatio-Temporal Alignment for Video Grounding

弥合时空:解耦的时空对齐用于视频定位

Xuezhen Tu, Jingyu Wu, Fangyu Kang, Qingpeng Nong, Kaijin Zhang, Chaoyue Niu, Fan Wu

机构 * Shanghai Jiao Tong University(上海交通大学) ZTE Corporation(中兴通讯)

专题命中 视频多模态 :MLLM(abstract,abstract_cn);multimodal(abstract);分类 cs.CV

AI总结 本文提出Bridge-STG框架,通过解耦时空定位并保持语义连贯性,解决视频定位中时空对齐和视觉token冗余问题,实验表明其在多个基准上达到SOTA性能。

详情

展开后加载摘要…

URL PDF HTML 收藏
2601.04567 2026-04-17 cs.CV 77%

All Changes May Have Invariant Principles: Improving Ever-Shifting Harmful Meme Detection via Design Concept Reproduction

所有变化可能都有不变原则:通过设计概念再现改进永动有害迷因检测

Ziyou Jiang, Mingyang Li, Junjie Wang, Yuekai Huang, Jie Huang, Zhiyuan Chang, Zhaoyang Li, Qing Wang

机构 * State Key Laboratory of Complex System Modeling and Simulation Technology(复杂系统建模与仿真技术国家重点实验室) Science and Technology on Integrated Information System Laboratory Institute of Software Chinese Academy of Sciences(软件研究所信息集成系统技术研究所) University of Chinese Academy of Sciences(中国科学院大学)

专题命中 视频多模态 :MLLM(abstract,abstract_cn);multimodal(abstract);分类 cs.CV

AI总结 本文提出RepMD方法,通过设计概念再现检测不断变化的有害迷因,利用攻击树定义设计概念图,并指导多模态大语言模型提高检测准确率。

Comments 19 pages, 11 figures, 9 tables accepted by ACL 2026 main conference

详情

展开后加载摘要…

URL PDF HTML 收藏
2604.11627 2026-04-14 cs.CV 77%

POINTS-Long: Adaptive Dual-Mode Visual Reasoning in MLLMs

POINTS-Long: 基于人类视觉系统的自适应双模式视觉推理MLLM

Haicheng Wang, Yuan Liu, Yikun Liu, Zhemeng Yu, Zhongyin Zhao, Yangxiu You, Zilin Yu, Le Tian, Xiao Zhou, Jie Zhou, Weidi Xie, Yanfeng Wang

机构 * SAI, Shanghai Jiao Tong University(上海交通大学人工智能研究院) WeChat AI, Tencent(腾讯微信人工智能)

专题命中 视频多模态 :multimodal(abstract);MLLM(abstract);cross-modal(abstract);分类 cs.CV

AI总结 POINTS-Long引入动态视觉token缩放机制,通过聚焦模式和待机模式实现效率与精度的动态平衡,在细粒度视觉任务中保持最优性能,在长形式视觉理解中使用1/40-1/10的视觉token保留97.7-99.7%的精度,支持流式视觉理解。

详情

展开后加载摘要…

URL PDF HTML 收藏
2512.24321 2026-01-01 cs.CV cs.RO 77%

UniAct: Unified Motion Generation and Action Streaming for Humanoid Robots

UniAct:面向人形机器人的统一动作生成与动作流

Nan Jiang, Zimo He, Wanhe Yu, Lexi Pang, Yunhao Li, Hongjie Li, Jieming Cui, Yuhan Li, Yizhou Wang, Yixin Zhu, Siyuan Huang

机构 * Institute for AI, Peking University(人工智能研究院,北京大学) Beijing Institute for General Artificial Intelligence (BIGAI)(北京通用人工智能研究院) School of Psychological and Cognitive Sciences, Peking University(心理学与认知科学学院,北京大学) School of Computer Science, Peking University(计算机科学学院,北京大学) Yuanpei College, Peking University(元培学院,北京大学) School of Foreign Languages, Peking University(外语学院,北京大学) School of EECS, Peking University(电子工程与科学学院,北京大学) Huazhong University of Science and Technology(华中科技大学) State Key Lab of General AI(通用人工智能国家重点实验室) Nat’l Eng. Research Center of Visual Technology(视觉技术国家工程研究中心) Beijing Key Laboratory of Behavior and Mental Health, Peking University(北京行为与心理健康重点实验室,北京大学) Embodied Intelligence Lab, PKU-Wuhan Institute for Artificial Intelligence(具身智能实验室,北京大学-武汉人工智能研究院)

专题命中 视频多模态 :multimodal(abstract);MLLM(abstract);cross-modal(abstract);分类 cs.CV

AI总结 UniAct通过统一感知与控制框架,实现人形机器人在多模态指令下的高效动作生成与实时执行,提升零样本跟踪成功率19%。

Comments Project page: https://jnnan.github.io/uniact/

详情

展开后加载摘要…

URL PDF HTML 收藏
2406.18113 2026-01-01 cs.CV 77%

Chrono: A Simple Blueprint for Representing Time in MLLMs

Chrono: 一种用于多模态大语言模型中表示时间的简单蓝图

Hector Rodriguez, Boris Meinardus, Anil Batra, Anna Rohrbach, Marcus Rohrbach

专题命中 视频多模态 :multimodal(abstract);MLLM(abstract);image-text(abstract);分类 cs.CV

AI总结 Chrono提出了一种简单通用的序列蓝图,用于提升多模态大语言模型在视频时间定位和 grounded 视频问答任务中的性能。

Comments Code: https://github.com/sudo-Boris/mr-Blip. Submitted to IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI). Under review

详情

展开后加载摘要…

URL PDF HTML 收藏
2507.00469 2025-07-02 cs.CV cs.LG 77%

Bisecle: Binding and Separation in Continual Learning for Video Language Understanding

Yue Tan, Xiaoqian Hu, Hao Xue, Celso De Melo, Flora D. Salim

机构 * School of Computer Science University of New South Wales(计算机科学学院新南威尔士大学) School of Computer Science and Engineering University of New South Wales(计算机科学与工程学院新南威尔士大学) DEVCOM Army Research Laboratory(陆军研究实验室)

专题命中 视频多模态 :multimodal(abstract);cross-modal(abstract);multimodal foundation model(abstract);分类 cs.CV

Comments 23 pages, 12 figures, 10 tables

详情

展开后加载摘要…

URL PDF HTML 收藏
2505.14231 2025-05-21 cs.CV 77%

UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning

Sule Bai, Mingxing Li, Yong Liu, Jing Tang, Haoji Zhang, Lei Sun, Xiangxiang Chu, Yansong Tang

机构 * Tsinghua Shenzhen International Graduate School, Tsinghua University(清华大学深圳国际研究生院) AMAP, Alibaba Group(阿里云研究院)

专题命中 视频多模态 :multimodal(abstract);multi-modal(abstract);MLLM(abstract);分类 cs.CV

详情

展开后加载摘要…

URL PDF HTML 收藏
2502.21257 2025-03-26 cs.RO cs.CV 77%

RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete

Yuheng Ji, Huajie Tan, Jiayu Shi, Xiaoshuai Hao, Yuan Zhang, Hengyuan Zhang, Pengwei Wang, Mengdi Zhao, Yao Mu, Pengju An, Xinda Xue, Qinghang Su, Huaihai Lyu, Xiaolong Zheng, Jiaming Liu, Zhongyuan Wang, Shanghang Zhang

专题命中 视频多模态 :multimodal(abstract);multi-modal(abstract);MLLM(abstract);分类 cs.CV

详情

展开后加载摘要…

URL PDF HTML 收藏
2411.11066 2024-11-19 cs.CV 77%

TS-LLaVA: Constructing Visual Tokens through Thumbnail-and-Sampling for Training-Free Video Large Language Models

Tingyu Qu, Mingxiao Li, Tinne Tuytelaars, Marie-Francine Moens

专题命中 视频多模态 :multimodal(abstract);multi-modal(abstract);image-text(abstract);分类 cs.CV

Comments work in progress

详情

展开后加载摘要…

URL PDF HTML 收藏
2406.06465 2024-06-11 cs.CV cs.AI cs.CL cs.LG cs.MM 77%

AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction

Zhen Xing, Qi Dai, Zejia Weng, Zuxuan Wu, Yu-Gang Jiang

专题命中 视频多模态 :multi-modal(abstract);MLLM(abstract);分类 cs.CV、cs.CL、cs.AI

详情

展开后加载摘要…

URL PDF HTML 收藏
2401.10588 2024-01-22 cs.CV 77%

DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval

Xiangpeng Yang, Linchao Zhu, Xiaohan Wang, Yi Yang

专题命中 视频多模态 :multi-modal(abstract);cross-modal(abstract);image-text(abstract);分类 cs.CV

Comments AAAI2024, Code will be available at https://github.com/knightyxp/DGL

详情

展开后加载摘要…

URL PDF HTML 收藏
2311.15075 2023-11-28 cs.CV 77%

Mug-STAN: Adapting Image-Language Pretrained Models for General Video Understanding

Ruyang Liu, Jingjia Huang, Wei Gao, Thomas H. Li, Ge Li

专题命中 视频多模态 :multimodal(abstract);multi-modal(abstract);image-text(abstract);分类 cs.CV

详情

展开后加载摘要…

URL PDF HTML 收藏
2209.07526 2022-10-21 cs.CV 77%

OmniVL:One Foundation Model for Image-Language and Video-Language Tasks

Junke Wang, Dongdong Chen, Zuxuan Wu, Chong Luo, Luowei Zhou, Yucheng Zhao, Yujia Xie, Ce Liu, Yu-Gang Jiang, Lu Yuan

专题命中 视频多模态 :multi-modal(abstract);cross-modal(abstract);image-text(abstract);分类 cs.CV

Comments To appear at NeurIPs 2022, Camera Ready with Typos fixed

详情

展开后加载摘要…

URL PDF HTML 收藏
2606.27187 2026-06-26 cs.CV cs.CL 新提交 76%

HarmVideoBench: Benchmarking Harmful Video Understanding in Large Multimodal Models

HarmVideoBench:大型多模态模型中有害视频理解的基准测试

Jiajun Wu, Haoyu Kang, Yining Sun, Jiacheng Hou, Heng Zhang, Danyang Zhang, Zhenjun Zhao, Haochi Zhang, Leixin Sun, Eric Hanchen Jiang, Yushan Li, Ruiyu Li, Mengkai Huang, Yan Gao, Xu Zhang, Guancheng Wan

机构 * Central South University(中南大学) Tsinghua University(清华大学) South China Normal University(华南师范大学) ByteDance Inc(字节跳动公司) University of Zaragoza(阿拉维达大学) CosmosMind Wuhan University(武汉大学) University of California, Los Angeles(加州大学洛杉矶分校) Southeast University(东南大学) Tencent(腾讯公司) Nankai University(南开大学) Supermicro Computer Inc(Supermicro计算机公司) Huazhong University of Science and Technology(华中科技大学)

专题命中 视频多模态 :multimodal(title);分类 cs.CV、cs.CL

AI总结 提出HarmVideoBench,一个包含1379个视频和4137道选择题的多层次诊断基准,从三个维度评估模型对有害视频的深层理解,并引入BCR方法将宏平均准确率从61.7%提升至84.4%。

详情

展开后加载摘要…

URL PDF HTML 收藏
2606.05917 2026-06-05 cs.CV cs.CL 76%

MemoryCard: Topic-Aware Multi-Modal Clue Compression for Long-Video Question Answering

MemoryCard: 面向长视频问答的主题感知多模态线索压缩

Qing Yang, Pengcheng Huang, Xinze Li, Zhenghao Liu, Yukun Yan, Yu Gu, Ge Yu, Gang Li, Maosong Sun

机构 * School of Computer Science and Engineering, Northeastern University(东北大学计算机科学与工程学院) Department of Computer Science and Technology, Tsinghua University(清华大学计算机科学与技术系) Digital China Group(数字中国集团)

专题命中 视频多模态 :multi-modal(title);分类 cs.CV、cs.CL

AI总结 提出MemoryCard框架,通过将长视频分割为主题事件单元并生成事件级摘要和代表性视觉时刻,以记忆卡形式增强VLMs的长视频问答能力,在相同视觉令牌预算下准确率提升高达21.8%。

Comments 21 pages, 8 figures

详情

展开后加载摘要…

URL PDF HTML 收藏
2602.24138 2026-05-21 cs.CV cs.AI 76%

Multimodal Optimal Transport for Training-free Temporal Segmentation in Surgical Robotics

多模态最优传输用于手术机器人中的无训练时序分割

Omar Mohamed, Edoardo Fazzari, Ayah Al-Naji, Hamdan Alhadhrami, Khalfan Hableel, Saif Alkindi, Ivan Laptev, Cesare Stefanini

机构 * Dept. of Robotics, Mohamed bin Zayed University of AI(机器人系,Mohamed bin Zayed人工智能大学)

专题命中 视频多模态 :multimodal(title);分类 cs.CV、cs.AI

AI总结 本文提出了一种无需标注的手术时序分割框架TASOT,通过结合时间对齐的文本描述和视觉信息,在统一的不平衡Gromov-Wasserstein最优传输目标下融合视觉和语义线索,实现了在多个公开手术数据集上的显著提升。

详情

展开后加载摘要…

URL PDF HTML 收藏
2304.11193 2026-05-14 cs.RO cs.AI cs.CV 76%

Multi-Modal World Model for Physical Robot Interactions: Simultaneous Visual and Tactile Predictions for Enhanced Accuracy

多模态世界模型用于物理机器人交互:同时进行视觉和触觉预测以提高准确性

Willow Mandil, Amir Ghalamzan-E

机构 * University of Lincoln(林肯大学) University of Sheffield(谢菲尔德大学)

专题命中 视频多模态 :multi-modal(title);分类 cs.CV、cs.AI

AI总结 本文提出多模态世界模型,通过整合视觉和触觉信息提升机器人物理交互的预测精度,特别是在物理模糊场景中表现更优。

Comments This paper is accepted for publication in Robotics and Autonomous Systems

详情

展开后加载摘要…

URL PDF HTML 收藏
2603.22911 2026-04-14 cs.CV cs.AI 76%

ForestPrune: High-ratio Visual Token Compression for Video Multimodal Large Language Models via Spatial-Temporal Forest Modeling

ForestPrune: 通过时空森林建模实现视频多模态大语言模型的高比率视觉令牌压缩

Shaobo Ju, Baiyang Song, Tao Chen, Jiapeng Zhang, Qiong Wu, Chao Chang, HuaiXi Wang, Yiyi Zhou, Rongrong Ji

机构 * Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University(厦门大学多媒体可信感知与高效计算教育部重点实验室) National University of Defense Technology(国防科技大学)

专题命中 视频多模态 :multimodal(title);分类 cs.CV、cs.AI

AI总结 本文提出ForestPrune,一种无需训练的视频MLLM令牌压缩方法,通过时空森林建模实现高效高比率压缩,实验表明其在视频任务中效果优于现有方法。

详情

展开后加载摘要…

URL PDF HTML 收藏
2504.16788 2026-03-16 cs.CV cs.AI 76%

Towards Explainable AI: Multi-Modal Transformer for Video-based Image Description Generation

迈向可解释AI:基于视频的图像描述生成的多模态Transformer

Lakshita Agarwal, Bindu Verma

专题命中 视频多模态 :multi-modal(title);分类 cs.CV、cs.AI

AI总结 本文提出一种多模态Transformer框架,结合文本和视觉模态生成视频描述,通过ResNet50提取视频特征并结合GPT-2模型,提升描述质量和可解释性。

Journal ref 2026

详情

展开后加载摘要…

URL PDF HTML 收藏
2410.01817 2025-07-23 cs.CV cs.AI cs.CY 76%

Aligning AI with Public Values: Deliberation and Decision-Making for Governing Multimodal LLMs in Political Video Analysis

Tanusree Sharma, Yujin Potter, Zachary Kilhoffer, Yun Huang, Dawn Song, Yang Wang

专题命中 视频多模态 :multimodal(title);分类 cs.CV、cs.AI

详情

展开后加载摘要…

URL PDF HTML 收藏
2506.21116 2025-07-09 cs.CV cs.AI 76%

IPFormer-VideoLLM: Enhancing Multi-modal Video Understanding for Multi-shot Scenes

Yujia Liang, Jile Jiao, Xuetao Feng, Zixuan Ye, Yuan Wang, Zhicheng Wang

机构 * School of AIA, Huazhong University of Science and Technology(华中科技大学人工智能学院) Deepeleph Intelligent Technology(DeepEleph智能技术) JD Explore Academy(JD探索学院) Department of Electronic Engineering, Tsinghua University(清华大学电子工程系)

专题命中 视频多模态 :multi-modal(title);分类 cs.CV、cs.AI

详情

展开后加载摘要…

URL PDF HTML 收藏
2412.07804 2025-03-06 eess.IV cs.AI cs.CV 76%

XLSTM-HVED: Cross-Modal Brain Tumor Segmentation and MRI Reconstruction Method Using Vision XLSTM and Heteromodal Variational Encoder-Decoder

Shenghao Zhu, Yifei Chen, Shuo Jiang, Weihong Chen, Chang Liu, Yuanhan Wang, Xu Chen, Yifan Ke, Feiwei Qin, Changmiao Wang, Zhu Zhu

专题命中 视频多模态 :cross-modal(title);分类 cs.CV、cs.AI

Comments 5 pages, 2 figures

Journal ref ISBI 2025

详情

展开后加载摘要…

URL PDF HTML 收藏
2501.16471 2025-01-29 cs.LG cs.AI eess.AS eess.IV q-bio.NC 76%

SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments

Simon Dahan, Gabriel Bénédict, Logan Z. J. Williams, Yourong Guo, Daniel Rueckert, Robert Leech, Emma C. Robinson

专题命中 视频多模态 :multimodal(title);分类 cs.AI、eess.AS

Comments 27 pages, accepted to ICLR 2025

详情

展开后加载摘要…

URL PDF HTML 收藏
2501.01422 2025-01-03 cs.CV cs.AI cs.LG 76%

Multi-Modal Video Feature Extraction for Popularity Prediction

Haixu Liu, Wenning Wang, Haoxiang Zheng, Penghao Jiang, Qirui Wang, Ruiqing Yan, Qiuzhuang Sun

专题命中 视频多模态 :multi-modal(title);分类 cs.CV、cs.AI

Comments INFORMS 2024 Data Challenge Competition

详情

展开后加载摘要…

URL PDF HTML 收藏
2412.06712 2024-12-10 cs.LG cs.CL cs.CV 76%

How to Merge Your Multimodal Models Over Time?

Sebastian Dziadzio, Vishaal Udandarao, Karsten Roth, Ameya Prabhu, Zeynep Akata, Samuel Albanie, Matthias Bethge

专题命中 视频多模态 :multimodal(title);分类 cs.CV、cs.CL

Comments Technical Report. Code at https://github.com/ExplainableML/fomo_in_flux

详情

展开后加载摘要…

URL PDF HTML 收藏
2411.03531 2024-11-07 cs.CV cs.AI 76%

Personalized Video Summarization by Multimodal Video Understanding

Brian Chen, Xiangyuan Zhao, Yingnan Zhu

专题命中 视频多模态 :multimodal(title);分类 cs.CV、cs.AI

Comments In Proceedings of CIKM 2024 Applied Research Track

Journal ref 33rd ACM International Conference on Information and Knowledge Management (CIKM 2024)

详情

展开后加载摘要…

URL PDF HTML 收藏
2108.10282 2024-10-28 cs.CV cs.AI cs.LG eess.IV 76%

ChiNet: Deep Recurrent Convolutional Learning for Multimodal Spacecraft Pose Estimation

Duarte Rondao, Nabil Aouf, Mark A. Richardson

专题命中 视频多模态 :multimodal(title);分类 cs.CV、cs.AI

Comments This work has been submitted to the IEEE for possible publication

详情

展开后加载摘要…

URL PDF HTML 收藏
2408.03340 2024-08-08 cs.MM cs.CV 76%

An Empirical Comparison of Video Frame Sampling Methods for Multi-Modal RAG Retrieval

Mahesh Kandhare, Thibault Gisselbrecht

专题命中 视频多模态 :multi-modal(title);分类 cs.CV、cs.MM

Comments 19 pages, 24 figures (65 images)

详情

展开后加载摘要…

URL PDF HTML 收藏