视觉大模型 / VLM - arXivDaily 专题

2408.07981 2024-08-16 cs.CV cs.AI 84%

LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning

Jiajie Li, Garrett Skinner, Gene Yang, Brian R Quaranto, Steven D Schwaitzberg, Peter C W Kim, Jinjun Xiong

专题命中 VLM训练与架构：LLaVA(title,abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

2407.19305 2024-08-08 cs.CV cs.LG q-bio.TO 84%

GP-VLS: A general-purpose vision language model for surgery

Samuel Schmidgall, Joseph Cho, Cyril Zakka, William Hiesinger

专题命中 VLM训练与架构：vision language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

2303.10571 2024-08-06 cs.LG cs.CV 84%

Reinforcement Learning Friendly Vision-Language Model for Minecraft

Haobin Jiang, Junpeng Yue, Hao Luo, Ziluo Ding, Zongqing Lu

专题命中 VLM训练与架构：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments ECCV 2024

2312.04533 2024-07-31 cs.RO cs.CV cs.LG 84%

Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models

Ivan Kapelyukh, Yifei Ren, Ignacio Alzugaray, Edward Johns

专题命中 VLM训练与架构：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments ICRA 2024. Project webpage with robot videos: https://www.robot-learning.uk/dream2real

2309.14928 2024-07-31 cs.CV cs.LG 84%

Noise-Tolerant Few-Shot Unsupervised Adapter for Vision-Language Models

Eman Ali, Muhammad Haris Khan

专题命中 VLM训练与架构：vision-language model(title,abstract)；vision language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted at BMVC 2024

2407.19185 2024-07-30 cs.CV cs.AI 84%

LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models

Ruiyi Zhang, Yufan Zhou, Jian Chen, Jiuxiang Gu, Changyou Chen, Tong Sun

专题命中 VLM训练与架构：LLaVA(title,abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2024 Under Review

2407.17428 2024-07-25 cs.CV cs.AI 84%

Vision Language Model-Empowered Contract Theory for AIGC Task Allocation in Teleoperation

Zijun Zhan, Yaxian Dong, Yuqing Hu, Shuai Li, Shaohua Cao, Zhu Han

专题命中 VLM训练与架构：vision language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments 11 pages, 10 figures

2404.02424 2024-06-26 cs.LG cs.CV 84%

Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration

Shwai He, Ang Li, Tianlong Chen

专题命中 VLM训练与架构：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

2306.16048 2024-06-19 cs.CV cs.AI 84%

Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity

Zhenlin Xu, Yi Zhu, Tiffany Deng, Abhay Mittal, Yanbei Chen, Manchen Wang, Paolo Favaro, Joseph Tighe, Davide Modolo

专题命中 VLM训练与架构：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments CVPR2024 MMFM workshop

2311.17647 2024-06-12 cs.CV cs.AI cs.CL 84%

Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?

Xiujun Li, Yujie Lu, Zhe Gan, Jianfeng Gao, William Yang Wang, Yejin Choi

专题命中 VLM训练与架构：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Github: https://github.com/VIM-Bench/VIM_TOOL, Model and Data: https://huggingface.co/VIM-Bench

2401.13201 2024-06-11 cs.CV cs.AI cs.CL 84%

MLLMReID: Multimodal Large Language Model-based Person Re-identification

Shan Yang, Yongfei Zhang

专题命中 VLM训练与架构：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

2406.02915 2024-06-06 cs.CV cs.LG 84%

Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models

Jinhao Li, Haopeng Li, Sarah Erfani, Lei Feng, James Bailey, Feng Liu

专题命中 VLM训练与架构：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments 22 pages, 16 figures, published to ICML 2024

2403.18715 2024-06-06 cs.CV cs.AI cs.CL cs.MM 84%

Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding

Xintong Wang, Jingheng Pan, Liang Ding, Chris Biemann

专题命中 VLM训练与架构：vision-language model(title,abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Comments Accepted to Findings of ACL 2024

2405.15684 2024-05-27 cs.CV cs.AI 84%

Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models

Yue Zhang, Hehe Fan, Yi Yang

专题命中 VLM训练与架构：multimodal large language model(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

2404.15406 2024-05-24 cs.CV cs.AI cs.CL cs.MM 84%

Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs

Davide Caffagni, Federico Cocchi, Nicholas Moratelli, Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

专题命中 VLM训练与架构：LLaVA(title,abstract)；visual question answering(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2024 Workshop on What is Next in Multimodal Foundation Models

2405.02246 2024-05-06 cs.CV cs.AI 84%

What matters when building vision-language models?

Hugo Laurençon, Léo Tronchon, Matthieu Cord, Victor Sanh

专题命中 VLM训练与架构：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

2404.11605 2024-04-18 cs.CV cs.AI cs.RO 84%

VG4D: Vision-Language Model Goes 4D Video Recognition

Zhichao Deng, Xiangtai Li, Xia Li, Yunhai Tong, Shen Zhao, Mengyuan Liu

专题命中 VLM训练与架构：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments ICRA 2024

2404.08856 2024-04-16 cs.CL cs.AI cs.LG 84%

On Speculative Decoding for Multimodal Large Language Models

Mukul Gagrani, Raghavv Goel, Wonseok Jeon, Junyoung Park, Mingu Lee, Christopher Lott

专题命中 VLM训练与架构：multimodal large language model(title,abstract)；LLaVA(abstract)；分类 cs.AI、cs.LG

Comments Accepted as a spotlight paper to ELVM workshop at CVPR 2024

2404.05183 2024-04-09 cs.CV cs.LG 84%

Progressive Alignment with VLM-LLM Feature to Augment Defect Classification for the ASE Dataset

Chih-Chung Hsu, Chia-Ming Lee, Chun-Hung Sun, Kuang-Ming Wu

专题命中 VLM训练与架构：VLM(title,abstract)；vision-language model(abstract)；分类 cs.CV、cs.LG

Comments MULA 2024

2403.18814 2024-03-28 cs.CV cs.AI cs.CL 84%

Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

Yanwei Li, Yuechen Zhang, Chengyao Wang, Zhisheng Zhong, Yixin Chen, Ruihang Chu, Shaoteng Liu, Jiaya Jia

专题命中 VLM训练与架构：vision language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Code and models are available at https://github.com/dvlab-research/MiniGemini

2403.09072 2024-03-15 cs.CV cs.AI cs.CL 84%

UniCode: Learning a Unified Codebook for Multimodal Large Language Models

Sipeng Zheng, Bohan Zhou, Yicheng Feng, Ye Wang, Zongqing Lu

专题命中 VLM训练与架构：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 14 pages, 2 figures, 11 tables

2402.02309 2024-02-07 cs.LG cs.CL cs.CR cs.CV 84%

Jailbreaking Attack against Multimodal Large Language Model

Zhenxing Niu, Haodong Ren, Xinbo Gao, Gang Hua, Rong Jin

专题命中 VLM训练与架构：multimodal large language model(title)；LLaVA(abstract)；MLLM(abstract)；分类 cs.CV、cs.LG

2401.12915 2024-01-24 cs.AI cs.CL cs.CV 84%

Red Teaming Visual Language Models

Mukai Li, Lei Li, Yuwei Yin, Masood Ahmed, Zhenguang Liu, Qi Liu

专题命中 VLM训练与架构：visual language model(title)；vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV、cs.AI

Comments Working in progress

2306.10169 2023-06-21 cs.CV cs.CL cs.LG 84%

Meta-Personalizing Vision-Language Models to Find Named Instances in Video

Chun-Hsiao Yeh, Bryan Russell, Josef Sivic, Fabian Caba Heilbron, Simon Jenni

专题命中 VLM训练与架构：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV、cs.LG

Comments Accepted to CVPR 2023. Project webpage: https://danielchyeh.github.io/metaper/

2606.28719 2026-06-30 cs.AI 84%

ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models

ComMem：视觉语言模型测试时自适应的互补记忆系统

Guanglong Sun, Shuang Cui, Bo Lei, Liyuan Wang, Zihan Zhai, Hongwei Yan, Hang Su, Jun Zhu, Yi Zhong

机构 * School of Life Sciences, IDG/McGovern Institute for Brain Research, Tsinghua University, Beijing, China（生命科学学院，IDG/麦克戈文脑科学研究院，清华大学，北京，中国）； Institute of Software Chinese Academy of Sciences, Beijing, China（中国科学院软件研究所，北京，中国）； Beijing Academy of Artificial Intelligence, Beijing, China（北京人工智能研究院，北京，中国）； Department of Psychological and Cognitive Sciences, Tsinghua University, Beijing, China（心理学与认知科学系，清华大学，北京，中国）； Dept. of Comp. Sci. and Tech., Institute for AI, Tsinghua-Bosch Joint ML Center, THBI Lab, BNRist Center, Tsinghua University, Beijing, China（计算机科学与技术系，人工智能研究院，清华大学-博世联合机器学习中心，THBI实验室，BNRist中心，清华大学，北京，中国）

专题命中 VLM训练与架构：vision-language model(title,abstract)；VLM(abstract_cn,comments)；分类 cs.AI

AI总结提出受生物互补记忆系统启发的ComMem方法，通过快速适应细节记忆和慢速整合抽象记忆协同工作，实现视觉语言模型在测试时的跨模态一致自适应，在15个基准数据集上显著优于现有方法。

Comments A brain-inspired complementary memory framework leveraging fast visual caching and slow textual refinement for VLM test-time adaptation

2511.08917 2026-04-01 cs.HC cs.CV 84%

"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models

由非残障人士训练的模型：评估图像质量如何影响产品描述生成

Kapil Garg, Xinru Tang, Jimin Heo, Dwayne R. Morgan, Darren Gergle, Erik B. Sudderth, Anne Marie Piper

机构 * University of California, Irvine（加州大学尔湾分校）； Northwestern University（西北大学）

专题命中 VLM训练与架构：vision-language model(title,abstract)；VLM(abstract,comments)；分类 cs.CV

AI总结研究评估图像质量对视觉语言模型生成产品描述准确性的影响，发现图像质量下降会导致模型性能显著降低，提出改进模型可靠性的建议。

Comments Published at CHI 2026; Honorable Mention for Best Paper (Top 5%). Dataset available at: https://github.com/Accessibility-Research-Collective-UCI/image-quality-vlm-chi26

2509.23661 2025-12-16 cs.CV 84%

LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training

LLaVA-OneVision-1.5：面向民主化多模态训练的完全开放框架

Xiang An, Yin Xie, Kaicheng Yang, Wenkang Zhang, Xiuwei Zhao, Zheng Cheng, Yirui Wang, Songcen Xu, Changrui Chen, Didi Zhu, Chunsheng Wu, Huajie Tan, Chunyuan Li, Jing Yang, Jie Yu, Xiyao Wang, Bin Qin, Yumeng Wang, Zizhen Yan, Ziyong Feng, Ziwei Liu, Bo Li, Jiankang Deng

机构 * LLaVA-OneVision Community Contributors（LLaVA-OneVision社区贡献者）

专题命中 VLM训练与架构：LLaVA(title,abstract)；vision-language model(abstract)；分类 cs.CV

AI总结 LLaVA-OneVision-1.5通过开放框架和高效训练方法实现了多模态模型的低成本高性能训练。

Comments LLaVA-OneVision-1.5 Technical Report

2510.10715 2025-10-14 cs.GR cs.CV 84%

VLM-Guided Adaptive Negative Prompting for Creative Generation

Shelly Golan, Yotam Nitzan, Zongze Wu, Or Patashnik

机构 * Adobe Research（Adobe研究院）； Tel Aviv University（特拉维夫大学）

专题命中 VLM训练与架构：VLM(title,abstract)；vision-language model(abstract)；分类 cs.CV

Comments Project page at: https://shelley-golan.github.io/VLM-Guided-Creative-Generation/

2412.09936 2024-12-16 cs.CV 84%

CaLoRAify: Calorie Estimation with Visual-Text Pairing and LoRA-Driven Visual Language Models

Dongyu Yao, Keling Yao, Junhong Zhou, Yinghao Zhang

专题命中 VLM训练与架构：visual language model(title)；vision-language model(abstract,comments)；VLM(abstract)；分类 cs.CV

Comments Disclaimer: This work is part of a course project and reflects ongoing exploration in the field of vision-language models and calorie estimation. Findings and conclusions are subject to further validation and refinement

2608.09778 2026-08-11 cs.RO 新提交 83%

RoboSeg: Online Part-Level Semantic Reconstruction for Robotic Manipulation via a Single Eye-in-Hand Camera

RoboSeg：基于单眼在手相机的机器人操作的在线部件级语义重建

Zhaochen Lan, Mengxiang Lin

机构 * School of Mechanical Engineering and Automation, Beihang University（北京航空航天大学机械工程及自动化学院）

专题命中 VLM训练与架构：VLM(summary_cn,abstract)；vision-language model(abstract)

AI总结 RoboSeg是无需CAD模型的部件级语义重建系统，结合VLM、TSDF融合与SAM3，实现机器人操作的部件语义索引，在24次物理试验中21次完成综合任务。

1. VLM训练与架构 5030 篇

LLaVA-Surg: Towards Multimodal Surgical Assistant via Structured Surgical Video Learning

GP-VLS: A general-purpose vision language model for surgery

Reinforcement Learning Friendly Vision-Language Model for Minecraft

Dream2Real: Zero-Shot 3D Object Rearrangement with Vision-Language Models

Noise-Tolerant Few-Shot Unsupervised Adapter for Vision-Language Models

LLaVA-Read: Enhancing Reading Ability of Multimodal Language Models

Vision Language Model-Empowered Contract Theory for AIGC Task Allocation in Teleoperation

Rethinking Pruning for Vision-Language Models: Strategies for Effective Sparsity and Performance Restoration

Benchmarking Zero-Shot Recognition with Vision-Language Models: Challenges on Granularity and Specificity

Text as Images: Can Multimodal Large Language Models Follow Printed Instructions in Pixels?

MLLMReID: Multimodal Large Language Model-based Person Re-identification

Visual-Text Cross Alignment: Refining the Similarity Score in Vision-Language Models

Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding

Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models

Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs

What matters when building vision-language models?

VG4D: Vision-Language Model Goes 4D Video Recognition

On Speculative Decoding for Multimodal Large Language Models

Progressive Alignment with VLM-LLM Feature to Augment Defect Classification for the ASE Dataset

Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models

UniCode: Learning a Unified Codebook for Multimodal Large Language Models

Jailbreaking Attack against Multimodal Large Language Model

Red Teaming Visual Language Models

Meta-Personalizing Vision-Language Models to Find Named Instances in Video

ComMem: Complementary Memory Systems for Test-Time Adaptation of Vision-Language Models

"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Models

LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training

VLM-Guided Adaptive Negative Prompting for Creative Generation

CaLoRAify: Calorie Estimation with Visual-Text Pairing and LoRA-Driven Visual Language Models

RoboSeg: Online Part-Level Semantic Reconstruction for Robotic Manipulation via a Single Eye-in-Hand Camera