视觉大模型 / VLM - arXivDaily 专题

2311.16714 2024-04-01 cs.CV 70%

Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, Yuhui Shi

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Accepted by CVPR 2024

2403.11835 2024-03-19 cs.CV 70%

Agent3D-Zero: An Agent for Zero-shot 3D Understanding

Sha Zhang, Di Huang, Jiajun Deng, Shixiang Tang, Wanli Ouyang, Tong He, Yanyong Zhang

专题命中 VLM训练与架构：VLM(abstract)；visual language model(abstract)；分类 cs.CV

Comments project page: https://zhangsha1024.github.io/Agent3D-Zero/

2403.01209 2024-03-05 cs.CV 70%

Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning

Shuo Yang, Zirui Shang, Yongqi Wang, Derong Deng, Hongwei Chen, Qiyuan Cheng, Xinxiao Wu

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2402.13232 2024-02-21 cs.CV cs.RO 70%

A Touch, Vision, and Language Dataset for Multimodal Alignment

Letian Fu, Gaurav Datta, Huang Huang, William Chung-Ho Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta, Roberto Calandra, Ken Goldberg

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2402.11690 2024-02-20 cs.CL cs.CV 70%

Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning

Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, Lifu Huang

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments 8 Pages, visual instruction tuning

2402.04764 2024-02-08 cs.LG 70%

Code as Reward: Empowering Reinforcement Learning with VLMs

David Venuto, Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, Ankit Anand

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.LG

2401.06807 2024-01-17 cs.CL cs.AI 70%

An EcoSage Assistant: Towards Building A Multimodal Plant Care Dialogue Assistant

Mohit Tomar, Abhisek Tiwari, Tulika Saha, Prince Jha, Sriparna Saha

专题命中 VLM训练与架构：VLM(abstract)；visual language model(abstract)；分类 cs.AI

2305.10400 2023-12-27 cs.CL cs.CV 70%

What You See is What You Read? Improving Text-Image Alignment Evaluation

Michal Yarom, Yonatan Bitton, Soravit Changpinyo, Roee Aharoni, Jonathan Herzig, Oran Lang, Eran Ofek, Idan Szpektor

专题命中 VLM训练与架构：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Accepted to NeurIPS 2023. Website: https://wysiwyr-itm.github.io/

2312.00351 2023-12-08 cs.CV 70%

Manipulating the Label Space for In-Context Classification

Haokun Chen, Xu Yang, Yuhang Huang, Zihan Wu, Jing Wang, Xin Geng

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2311.15812 2023-11-28 cs.CV 70%

C-SAW: Self-Supervised Prompt Learning for Image Generalization in Remote Sensing

Avigyan Bhattacharya, Mainak Singha, Ankit Jha, Biplab Banerjee

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Accepted in ACM ICVGIP 2023

2310.13596 2023-10-23 cs.CL cs.AI 70%

MarineGPT: Unlocking Secrets of Ocean to the Public

Ziqiang Zheng, Jipeng Zhang, Tuan-Anh Vu, Shizhe Diao, Yue Him Wong Tim, Sai-Kit Yeung

专题命中 VLM训练与架构：vision-language model(abstract)；MLLM(abstract)；分类 cs.AI

Comments work in progress. Code and data will be available at https://github.com/hkust-vgd/MarineGPT

2309.14525 2023-09-27 cs.CV cs.CL 70%

Aligning Large Multimodal Models with Factually Augmented RLHF

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell

专题命中 VLM训练与架构：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV

Comments Preprint

2306.15195 2023-07-04 cs.CV 70%

Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, Rui Zhao

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

2306.04387 2023-06-09 cs.CV cs.CL 70%

M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning

Lei Li, Yuwei Yin, Shicheng Li, Liang Chen, Peiyi Wang, Shuhuai Ren, Mukai Li, Yazheng Yang, Jingjing Xu, Xu Sun, Lingpeng Kong, Qi Liu

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Fix dataset url: https://huggingface.co/datasets/MMInstruction/M3IT Project: https://m3-it.github.io/

2306.00310 2023-06-02 cs.CV 70%

Prompt Algebra for Task Composition

Pramuditha Perera, Matthew Trager, Luca Zancato, Alessandro Achille, Stefano Soatto

专题命中 VLM训练与架构：VLM(abstract)；visual language model(abstract)；分类 cs.CV

2303.12417 2023-03-28 cs.CV 70%

CLIP$^2$: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data

Yihan Zeng, Chenhan Jiang, Jiageng Mao, Jianhua Han, Chaoqiang Ye, Qingqiu Huang, Dit-Yan Yeung, Zhen Yang, Xiaodan Liang, Hang Xu

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments To appear at CVPR 2023

2302.12066 2023-02-24 cs.CV 70%

Teaching CLIP to Count to Ten

Roni Paiss, Ariel Ephrat, Omer Tov, Shiran Zada, Inbar Mosseri, Michal Irani, Tali Dekel

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2112.08723 2022-10-18 cs.CL cs.CV 70%

Distilled Dual-Encoder Model for Vision-Language Understanding

Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, Furu Wei

专题命中 VLM训练与架构：visual reasoning(abstract)；visual question answering(abstract)；分类 cs.CV

Comments EMNLP 2022

2208.10442 2022-09-01 cs.CV cs.CL 70%

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Wenhui Wang, Hangbo Bao, Li Dong, Johan Bjorck, Zhiliang Peng, Qiang Liu, Kriti Aggarwal, Owais Khan Mohammed, Saksham Singhal, Subhojit Som, Furu Wei

专题命中 VLM训练与架构：visual reasoning(abstract)；visual question answering(abstract)；分类 cs.CV

Comments 18 pages

2203.14395 2022-07-29 cs.CV 70%

Single-Stream Multi-Level Alignment for Vision-Language Pretraining

Zaid Khan, Vijay Kumar BG, Xiang Yu, Samuel Schulter, Manmohan Chandraker, Yun Fu

专题命中 VLM训练与架构：visual question answering(abstract)；grounding(abstract)；分类 cs.CV

Comments ECCV 2022

2310.03744 2024-05-17 cs.CV cs.AI cs.CL cs.LG 69%

Improved Baselines with Visual Instruction Tuning

Haotian Liu, Chunyuan Li, Yuheng Li, Yong Jae Lee

专题命中 VLM训练与架构：LLaVA(abstract,comments)；分类 cs.CV、cs.AI、cs.LG

Comments Camera ready, CVPR 2024 (highlight). LLaVA project page: https://llava-vl.github.io

2304.08485 2023-12-14 cs.CV cs.AI cs.CL cs.LG 69%

Visual Instruction Tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee

专题命中 VLM训练与架构：LLaVA(abstract,comments)；分类 cs.CV、cs.AI、cs.LG

Comments NeurIPS 2023 Oral; project page: https://llava-vl.github.io/

2608.17453 2026-08-21 cs.RO 版本更新 67%

EATR-Stereo: Embodiment-Aware Token Routing of Paired Stereo Evidence for Humanoid Vision-Language-Action Control

EATR-Stereo：面向人形机器人视觉-语言-动作控制的具身感知配对立体证据路由

Songwei Wu, Rui Zhao, Fan Yang, Zhongqiang Nie, Zhiduo Jiang, Wandong Sun, Yuwei Li, Jian Hu, Yang Liu, Hong Liu

机构 * Harbin Institute of Technology（哈尔滨工业大学）

专题命中 VLM训练与架构：grounding(abstract,abstract_cn)

AI总结 EATR-Stereo是具身感知的令牌路由框架，在冻结预训练VLA模型的前提下，通过选择性路由配对立体证据，提升了人形机器人长时程视觉-语言-动作控制的任务成功率。

Comments 8 pages, 5 figures

2608.14763 2026-08-18 eess.IV cs.AI cs.CV cs.LG 新提交 67%

Cross-Modal Ultrasound-MRI Learning for Fetal Brain Ventricular Volumetry and Abnormality Screening

用于胎儿脑室体积测量与异常筛查的跨模态超声-MRI学习

Yuhao Huang, Yuanji Zhang, Yuhuan Lu, Dong Ni, P. Ellen Grant, Davood Karimi

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

AI总结本研究提出跨模态学习框架VIFBA，基于超声视频实现胎儿脑室体积预测、VM严重程度分类及非VM异常筛查，性能优于基线与现有模型，为产前脑筛查提供实用方案。

Comments 17 pages, 11 figures, 6 tables

2608.15269 2026-08-18 cs.RO 新提交 67%

Remember Smarter: Visual History Compressor and Hyperbolic Experience Space for Robotic Memory

更智能地记忆：用于机器人记忆的视觉历史压缩器与双曲经验空间

Dai Zhou, Jiexi Yan, Tong Li, Yuxuan Wang, Cheng Deng

机构 * Xidian University（西安电子科技大学）

专题命中 VLM训练与架构：VLM(abstract,abstract_cn)

AI总结该研究提出即插即用模块 RS，通过双分支结构压缩视觉历史并组织经验，适配 pi0 后显著提升机器人任务成功率，在真实机器人实验中表现优异。

Comments 19 pages, 7 pages

2602.05408 2026-08-18 cs.IR 版本更新 67%

Rich-Media Re-Ranker: A User Satisfaction-Driven LLM Re-ranking Framework for Rich-Media Search

多介质重排序：一种基于用户满意度的LLM重排序框架用于多介质搜索

Zihao Guo, Ligang Zhou, Zeyang Tang, Feicheng Li, Ying Nie, Zhiming Peng, Qingyun Sun, Jianxin Li

专题命中 VLM训练与架构：VLM(abstract,abstract_cn)

AI总结本文提出Rich-Media Re-Ranker框架，通过多维细粒度建模提升用户搜索满意度，整合丰富侧信息和视觉信号，通过多任务强化学习增强系统适应性，实验证明其优于现有方法。

Comments Accepted by the Full Research Track of CIKM-2026

2605.08384 2026-08-18 cs.CL 版本更新 67%

jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers

jina-embeddings-v5-omni: 通过锁定对齐塔实现几何保持嵌入

Florian Hönicke, Michael Günther, Andreas Koukounas, Mohammad Kalim Akram, Saba Sturua, Han Xiao

机构 * Jina by Elastic（Jina 由 Elastic 公司）

专题命中 VLM训练与架构：VLM(abstract,abstract_cn)

AI总结本文提出GELATO方法，通过冻结对齐塔实现多模态嵌入，生成统一语义空间，训练效率高且保持文本嵌入一致性。

Comments 11 pages, 9 figures, 5 tables

2606.14061 2026-08-04 cs.SE 版本更新 67%

LLM Agents Can See Code Repositories

LLM智能体能够查看代码仓库

Dongjian Ma, Silin Chen, Yufei Yang, Yuling Shi, Yanfu Yan, Xiaodong Gu

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract_cn)

AI总结研究多模态大语言模型在仓库级问题解决中利用视觉表示的效果，发现纯视觉方案降低精度，而结合视觉结构图可减少26%输入token并保持或提升精度。

Comments Accepted by ASE 2026. Our code and data are available at https://github.com/cslsolow/SeeRepo

2607.25948 2026-07-29 cs.CV cs.AI cs.LG 新提交 67%

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

MODUS：仅解码器的多模态任意到任意建模

Mingqiao Ye, Zhaochong An, Zhitong Gao, Xian Liu, François Fleuret, Chuan Li, Amir Zadeh, Serge Belongie, Afshin Dehghan, Jesse Allardice, David Mizrahi, Oğuzhan Fatih Kar, Roman Bachmann, Amir Zamir

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

AI总结研究任意到任意多模态建模，提出仅解码器的对称多模态建模方法Modus，无需特定模态组件，支持多种应用，在各基准测试中用单模型展现强大性能且与基线竞争，材料开源。

Comments Accepted at ICML 2026. Project page: https://modus-multimodal.epfl.ch

2602.19149 2026-07-28 cs.CR 版本更新 67%

ReVision : A Post-Hoc, Vision-Based Technique for Replacing Unacceptable Concepts in Image Generation Pipeline

ReVision：一种基于视觉的后处理技术，用于在图像生成管道中替换不可接受的概念

Gurjot Singh, Prabhjot Singh, Aashima Sharma, Maninder Singh, Ryan Ko

专题命中 VLM训练与架构：VLM(abstract,abstract_cn)

AI总结 ReVision是一种无需训练的后处理框架，通过视觉模型检测并替换图像生成中不安全的概念，提升安全性与生成质量。

1. VLM训练与架构 5058 篇

Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

Agent3D-Zero: An Agent for Zero-shot 3D Understanding

Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning

A Touch, Vision, and Language Dataset for Multimodal Alignment

Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning

Code as Reward: Empowering Reinforcement Learning with VLMs

An EcoSage Assistant: Towards Building A Multimodal Plant Care Dialogue Assistant

What You See is What You Read? Improving Text-Image Alignment Evaluation

Manipulating the Label Space for In-Context Classification

C-SAW: Self-Supervised Prompt Learning for Image Generalization in Remote Sensing

MarineGPT: Unlocking Secrets of Ocean to the Public

Aligning Large Multimodal Models with Factually Augmented RLHF

Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic

M$^3$IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuning

Prompt Algebra for Task Composition

CLIP$^2$: Contrastive Language-Image-Point Pretraining from Real-World Point Cloud Data

Teaching CLIP to Count to Ten

Distilled Dual-Encoder Model for Vision-Language Understanding

Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks

Single-Stream Multi-Level Alignment for Vision-Language Pretraining

Improved Baselines with Visual Instruction Tuning

Visual Instruction Tuning

EATR-Stereo: Embodiment-Aware Token Routing of Paired Stereo Evidence for Humanoid Vision-Language-Action Control

Cross-Modal Ultrasound-MRI Learning for Fetal Brain Ventricular Volumetry and Abnormality Screening

Remember Smarter: Visual History Compressor and Hyperbolic Experience Space for Robotic Memory

Rich-Media Re-Ranker: A User Satisfaction-Driven LLM Re-ranking Framework for Rich-Media Search

jina-embeddings-v5-omni: Geometry-preserving Embeddings via Locked Aligned Towers

LLM Agents Can See Code Repositories

MODUS: Decoder-Only Any-to-Any Modeling of Diverse Modalities

ReVision : A Post-Hoc, Vision-Based Technique for Replacing Unacceptable Concepts in Image Generation Pipeline