视觉大模型 / VLM

Enhancing Large Multimodal Models with Adaptive Sparsity and KV Cache Compression

Te Zhang, Yuheng Li, Junxiang Wang, Lujun Li

机构 * University of Michigan（密歇根大学） ； Johns Hopkins University（约翰霍普金斯大学） ； Central South university（中南大学） ； HKGAI 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.AI、cs.LG

Comments 6 pages

CopyJudge: Automated Copyright Infringement Identification and Mitigation in Text-to-Image Diffusion Models

Shunchang Liu, Zhuan Shi, Lingjuan Lyu, Yaochu Jin, Boi Faltings

机构 * EPFL（苏黎世联邦理工学院） ； Mila - Quebec AI Institute Mcgill University（蒙特利尔麦吉尔大学人工智能研究所） ； Westlake University（西湖大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ACM MM 2025

Efficient Learning for Product Attributes with Compact Multimodal Models

Mandar Kulkarni

机构 * Flipkart Data Science（Flipkart数据科学） 纠错

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.CV、cs.AI

Step-3 is Large yet Affordable: Model-system Co-design for Cost-effective Decoding

StepFun, :, Bin Wang, Bojun Wang, Changyi Wan, Guanzhe Huang, Hanpeng Hu, Haonan Jia, Hao Nie, Mingliang Li, Nuo Chen, Siyu Chen, Song Yuan, Wuxun Xie, Xiaoniu Song, Xing Chen, Xingping Yang, Xuelin Zhang, Yanbo Yu, Yaoyu Wang, Yibo Zhu, Yimin Jiang, Yu Zhou, Yuanwei Lu, Houyi Li, Jingcheng Hu, Ka Man Lo, Ailin Huang, Binxing Jiao, Bo Li, Boyu Chen, Changxin Miao, Chang Lou, Chen Hu, Chen Xu, Chenfeng Yu, Chengyuan Yao, Daokuan Lv, Dapeng Shi, Deshan Sun, Ding Huang, Dingyuan Hu, Dongqing Pang, Enle Liu, Fajie Zhang, Fanqi Wan, Gulin Yan, Han Zhang, Han Zhou, Hanghao Wu, Hangyu Guo, Hanqi Chen, Hanshan Zhang, Hao Wu, Haocheng Zhang, Haolong Yan, Haoran Lv, Haoran Wei, Hebin Zhou, Heng Wang, Heng Wang, Hongxin Li, Hongyu Zhou, Hongyuan Wang, Huiyong Guo, Jia Wang, Jiahao Gong, Jialing Xie, Jian Zhou, Jianjian Sun, Jiaoren Wu, Jiaran Zhang, Jiayu Liu, Jie Cheng, Jie Luo, Jie Yan, Jie Yang, Jieyi Hou, Jinguang Zhang, Jinlan Cao, Jisheng Yin, Junfeng Liu, Junhao Huang, Junzhe Lin, Kaijun Tan, Kaixiang Li, Kang An, Kangheng Lin, Kenkun Liu, Lei Yang, Liang Zhao, Liangyu Chen, Lieyu Shi, Liguo Tan, Lin Lin, Lin Zhang, Lina Chen, Liwen Huang, Liying Shi, Longlong Gu, Mei Chen, Mengqiang Ren, Ming Li, Mingzhe Chen, Na Wang, Nan Wu, Qi Han, Qian Zhao, Qiang Zhang, Qianni Liu, Qiaohui Chen, Qiling Wu, Qinglin He, Qinyuan Tan, Qiufeng Wang, Qiuping Wu, Qiuyan Liang, Quan Sun, Rui Li, Ruihang Miao, Ruosi Wan, Ruyan Guo, Shangwu Zhong, Shaoliang Pang, Shengjie Fan, Shijie Shang, Shilei Jiang, Shiliang Yang, Shiming Hao, Shuli Gao, Siming Huang, Siqi Liu, Tiancheng Cao, Tianhao Cheng, Tianhao Peng, Wang You, Wei Ji, Wen Sun, Wenjin Deng, Wenqing He, Wenzhen Zheng, Xi Chen, Xiangwen Kong, Xianzhen Luo, Xiaobo Yang, Xiaojia Liu, Xiaoxiao Ren, Xin Han, Xin Li, Xin Wu, Xu Zhao, Yanan Wei, Yang Li, Yangguang Li, Yangshijie Xu, Yanming Xu, Yaqiang Shi, Yeqing Shen, Yi Yang, Yifei Yang, Yifeng Gong, Yihan Chen, Yijing Yang, Yinmin Zhang, Yizhuang Zhou, Yuanhao Ding, Yuantao Fan, Yuanzhen Yang, Yuchu Luo, Yue Peng, Yufan Lu, Yuhang Deng, Yuhe Yin, Yujie Liu, Yukun Chen, Yuling Zhao, Yun Mou, Yunlong Li, Yunzhou Ju, Yusheng Li, Yuxiang Yang, Yuxiang Zhang, Yuyang Chen, Zejia Weng, Zhe Xie, Zheng Ge, Zheng Gong, Zhenyi Lu, Zhewei Huang, Zhichao Chang, Zhiguo Huang, Zhirui Wang, Zidong Yang, Zili Wang, Ziqi Wang, Zixin Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Xiangyu Zhang

机构 * StepFun Inc（StepFun公司） 纠错

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.AI、cs.LG

OS-MAP: How Far Can Computer-Using Agents Go in Breadth and Depth?

Xuetian Chen, Yinghao Chen, Xinfeng Yuan, Zhuo Peng, Lu Chen, Yuekeng Li, Zhoujia Zhang, Yingqian Huang, Leyan Huang, Jiaqing Liang, Tianbao Xie, Zhiyong Wu, Qiushi Sun, Biqing Qi, Bowen Zhou

机构 * Fudan University（复旦大学） ； Shanghai AI Lab（上海人工智能实验室） ； Tsinghua University（清华大学） ； The University of Hong Kong（香港大学） 纠错

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI

Comments Work in progress

Steering into New Embedding Spaces: Analyzing Cross-Lingual Alignment Induced by Model Interventions in Multilingual Language Models

Anirudh Sundar, Sinead Williamson, Katherine Metcalf, Barry-John Theobald, Skyler Seto, Masha Fedzechkina

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.AI、cs.LG

Comments 34 pages

CheXPO: Preference Optimization for Chest X-ray VLMs with Counterfactual Rationale

Xiao Liang, Jiawei Hu, Di Wang, Zhi Ma, Lin Zhao, Ronghan Li, Bo Wan, Quan Wang

机构 * The Key Laboratory of Smart Human-Computer Interaction（智能人机交互关键实验室） ； Wearable Technology of Shaanxi Province, Xidian University, Xi'an, China（陕西 wearable 技术, 西安电子科技大学, 中国） ； Nanjing University of Science and Technology（南京理工大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Breach in the Shield: Unveiling the Vulnerabilities of Large Language Models

Runpeng Dai, Run Yang, Fan Zhou, Hongtu Zhu

机构 * University of North Carolina at Chapel Hill（北卡罗来纳大学教堂山分校） ； BiliBili（哔哩哔哩） ； Shanghai University of Finance and Economics（上海财经大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.AI、cs.LG

Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for Radiology Report Generation

Xi Zhang, Zaiqiao Meng, Jake Lever, Edmond S. L. Ho

机构 * School of Computing Science, University of Glasgow（计算科学学院，格拉斯哥大学） 纠错

专题命中 VLM训练与架构 ：visual language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted by BioNLP@ACL 2024

Journal ref Proceedings of the 23rd Workshop on Biomedical Natural Language Processing, ACL 2024, pp. 624-634

Exploring Object Status Recognition for Recipe Progress Tracking in Non-Visual Cooking

Franklin Mingzhe Li, Kaitlyn Ng, Bin Zhu, Patrick Carrington

机构 * Carnegie Mellon University（卡内基梅隆大学） ； Singapore Management University（新加坡管理学院） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments ASSETS 2025

MGAA: Multi-Granular Adaptive Allocation fof Low-Rank Compression of LLMs

Guangyan Li, Yongqiang Tang, Wensheng Zhang

机构 * State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所） ； School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院） ； School of Computer Science and Cyber Engineering, Guangzhou University（广州大学计算机科学与网络工程学院） 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.AI、cs.LG

Comments 13 pages, 8 figures

Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models

Aarti Ghatkesar, Ganesh Venkatesh

机构 * AppliedML, Cerebras（应用机器学习，Cerebras） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Bisecle: Binding and Separation in Continual Learning for Video Language Understanding

Yue Tan, Xiaoqian Hu, Hao Xue, Celso De Melo, Flora D. Salim

机构 * School of Computer Science University of New South Wales（计算机科学学院新南威尔士大学） ； School of Computer Science and Engineering University of New South Wales（计算机科学与工程学院新南威尔士大学） ； DEVCOM Army Research Laboratory（陆军研究实验室） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments 23 pages, 12 figures, 10 tables

Ovis-U1 Technical Report

Guo-Hua Wang, Shanshan Zhao, Xinjie Zhang, Liangfu Cao, Pengxin Zhan, Lunhao Duan, Shiyin Lu, Minghao Fu, Xiaohao Chen, Jianshan Zhao, Yang Li, Qing-Guo Chen

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.CV、cs.AI

Comments An unified model for multimodal understanding, text-to-image generation, and image editing. GitHub: https://github.com/AIDC-AI/Ovis-U1

SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuning

Ziqi Wang, Chang Che, Qi Wang, Yangyang Li, Zenglin Shi, Meng Wang

机构 * Hefei University of Technology（合肥工业大学） ； Tsinghua University（清华大学） ； Academy of Cyber（网络学院） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

DriveBLIP2: Attention-Guided Explanation Generation for Complex Driving Scenarios

Shihong Ling, Yue Wan, Xiaowei Jia, Na Du

机构 * School of Computing and Information, University of Pittsburgh（计算与信息学院，匹兹堡大学） 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted to IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2025. 7 pages, 3 figures

Video-XL-2: Towards Very Long-Video Understanding Through Task-Aware KV Sparsification

Minghao Qin, Xiangrui Liu, Zhengyang Liang, Yan Shu, Huaying Yuan, Juenjie Zhou, Shitao Xiao, Bo Zhao, Zheng Liu

机构 * Beijing Academy of Artificial Intelligence（北京人工智能研究院） ； Shanghai Jiao Tong University（上海交通大学） ； University of Trento（特伦特大学） ； Renmin University of China（中国人民大学） ； Beijing University of Posts and Telecommunications（北京邮电大学） ； Hong Kong Polytechnic University（香港理工大学） 纠错

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.CV、cs.AI

Comments 12 pages, 5 Figure, 3 Table

How Visual Representations Map to Language Feature Space in Multimodal LLMs

Constantin Venhoff, Ashkan Khakzar, Sonia Joseph, Philip Torr, Neel Nanda

机构 * University of Oxford（牛津大学） ； McGill University（麦吉尔大学） ； Meta 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

SAE-V: Interpreting Multimodal Models for Enhanced Alignment

Hantao Lou, Changye Li, Jiaming Ji, Yaodong Yang

机构 * Institute for AI, Peking University, Beijing, China（人工智能研究院，北京大学，北京，中国） ； State Key Laboratory of General Artificial Intelligence, Institute for AI, Peking University, Beijing, China（通用人工智能国家重点实验室，人工智能研究院，北京大学，北京，中国） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments 17 pages, 13 figures

LARP: Tokenizing Videos with a Learned Autoregressive Generative Prior

Hanyu Wang, Saksham Suri, Yixuan Ren, Hao Chen, Abhinav Shrivastava

机构 * University of Maryland, College Park（马里兰大学 College Park 分校） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments ICLR 2025. Project page: https://hywang66.github.io/larp/

TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learning

Junru Zhang, Lang Feng, Xu Guo, Yuhan Wu, Yabo Dong, Duanqing Xu

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments Preprint

ROSA: Harnessing Robot States for Vision-Language and Action Alignment

Yuqing Wen, Kefan Gu, Haoxuan Liu, Yucheng Zhao, Tiancai Wang, Haoqiang Fan, Xiaoyan Sun

机构 * University of Science and Technology of China（中国科学技术大学） ； Nanjing University（南京大学） ； Dexmal 纠错

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Unifying Specialized Visual Encoders for Video Language Models

Jihoon Chung, Tyler Zhu, Max Gonzalez Saez-Diez, Juan Carlos Niebles, Honglu Zhou, Olga Russakovsky

机构 * Department of Computer Science, Princeton University, Princeton, NJ, United States（普林斯顿大学计算机科学系） ； Salesforce Research, Palo Alto, CA, United States（Salesforce研究） 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.LG

Comments Accepted to ICML 2025 as a Poster. Project page: https://tylerzhu.com/merv/

RONA: Pragmatically Diverse Image Captioning with Coherence Relations

Aashish Anantha Ramakrishnan, Aadarsh Anantha Ramakrishnan, Dongwon Lee

机构 * The Pennsylvania State University（宾夕法尼亚州立大学） ； National Institute of Technology, Tiruchirappalli（特里奇里帕利理工学院） 纠错

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted in the NAACL Fourth Workshop on Intelligent and Interactive Writing Assistants (In2Writing), Albuquerque, New Mexico, May 2025, https://in2writing.glitch.me

Enhancing Few-Shot Vision-Language Classification with Large Multimodal Model Features

Chancharik Mitra, Brandon Huang, Tianning Chai, Zhiqiu Lin, Assaf Arbelle, Rogerio Feris, Leonid Karlinsky, Trevor Darrell, Deva Ramanan, Roei Herzig

机构 * Carnegie Mellon University（卡内基梅隆大学） ； University of California, Berkeley（加州大学伯克利分校） ； IBM Research（IBM研究院） ； MIT-IBM Watson AI Lab（麻省理工-IBM沃森人工智能实验室） 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI

ARMOR: Empowering Multimodal Understanding Model with Interleaved Multimodal Generation Capability

Jianwen Sun, Yukang Feng, Chuanhao Li, Fanrui Zhang, Zizhen Li, Jiaxin Ai, Sizhuo Zhou, Yu Dai, Shenglin Zhang, Kaipeng Zhang

机构 * Nankai University（南开大学） ； Shanghai Innovation Institute（上海创新研究院） ； University of Science and Technology of China（中国科学技术大学） ； Wuhan University（武汉大学） ； Shanghai AI Laboratory（上海人工智能实验室） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Line of Sight: On Linear Representations in VLLMs

Achyuta Rajaram, Sarah Schwettmann, Jacob Andreas, Arthur Conmy

机构 * MIT CSAIL（麻省理工学院计算机科学与人工智能实验室） 纠错

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI

Comments 8 pages, 9 figures

MINT: Memory-Infused Prompt Tuning at Test-time for CLIP

Jiaming Yi, Ruirui Pan, Jishen Yang, Xiulong Yang

机构 * School of Computer, Central China Normal University（中央财经大学计算机学院） ； Amazon（亚马逊） 纠错

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI

Comments 14 pages, 3 figures

RACE-Align: Retrieval-Augmented and Chain-of-Thought Enhanced Preference Alignment for Large Language Models

Qihang Yan, Xinyu Zhang, Luming Guo, Qi Zhang, Feifan Liu

机构 * ShanghaiTech University（上海科技大学） ； Henan University（河南大学） ； Liaoning University of Traditional Chinese Medicine（辽宁中医药大学） 纠错

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.AI、cs.LG

Curriculum Learning with Quality-Driven Data Selection

Biao Wu, Ling Chen

机构 * Australian Artificial Intelligence Institute（澳大利亚人工智能研究所） 纠错

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG