视觉大模型 / VLM

The Dark Side of Rich Rewards: Understanding and Mitigating Noise in VLM Rewards

Sukai Huang, Shu-Wei Liu, Nir Lipovetzky, Trevor Cohn

机构 * Google DeepMind（谷歌DeepMind） 纠错

专题命中 GUI与屏幕智能体 ：VLM(title,abstract)；vision-language model(abstract)；分类 cs.LG

Comments accepted by PRL Workshop Series @ ICAPS 2025. 11 main body pages, 21 appendix pages

AquaVLM: Improving Underwater Situation Awareness with Mobile Vision Language Models

Beitong Tian, Lingzhi Zhao, Bo Chen, Haozhen Zheng, Jingcheng Yang, Mingyuan Wu, Deepak Vasisht, Klara Nahrstedt

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(title)；vision-language model(abstract)；VLM(abstract)；分类 cs.AI

Comments 12 pages, 10 figures, under review

From Code to Action: Hierarchical Learning of Diffusion-VLM Policies

Markus Peschl, Pietro Mazzaglia, Daniel Dijkman

机构 * Qualcomm AI Research（高通人工智能研究） 纠错

专题命中 GUI与屏幕智能体 ：VLM(title,abstract)；vision-language model(abstract)；分类 cs.LG

Comments 19 pages including references, 6 figures. Accepted to CoRL LEAP 2025

Poison Once, Control Anywhere: Clean-Text Visual Backdoors in VLM-based Mobile Agents

Xuan Wang, Siyuan Liang, Zhe Liu, Yi Yu, Aishan Liu, Yuliang Lu, Xitong Gao, Ee-Chien Chang

专题命中 GUI与屏幕智能体 ：VLM(title,abstract)；vision-language model(abstract)；分类 cs.AI

Comments 10 pages

DivScene: Towards Open-Vocabulary Object Navigation with Large Vision Language Models in Diverse Scenes

Zhaowei Wang, Hongming Zhang, Tianqing Fang, Ye Tian, Yue Yang, Kaixin Ma, Xiaoman Pan, Yangqiu Song, Dong Yu

机构 * CSE Department, HKUST（香港科技大学计算机科学与工程系） ； Tencent AI Lab（腾讯AI实验室） ； Robotics X, Tencent（腾讯机器人实验室） ； University of Pennsylvania（宾夕法尼亚大学） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(title)；vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments EMNLP 2025

PerPilot: Personalizing VLM-based Mobile Agents via Memory and Exploration

Xin Wang, Zhiyao Cui, Hao Li, Ya Zeng, Chenxu Wang, Ruiqi Song, Yihang Chen, Kun Shao, Qiaosheng Zhang, Jinzhuo Liu, Siyue Ren, Shuyue Hu, Zhen Wang

专题命中 GUI与屏幕智能体 ：VLM(title,abstract)；vision language model(abstract)；分类 cs.AI

Utilizing Vision-Language Models as Action Models for Intent Recognition and Assistance

Cesar Alan Contreras, Manolis Chiou, Alireza Rastegarpanah, Michal Szulik, Rustam Stolkin

机构 * University of Birmingham（伯明翰大学） ； Queen Mary University of London（伦敦大学女王学院） ； Aston University（阿斯顿大学） ； United Kingdom National Nuclear Laboratory Ltd.（英国国家核实验室有限公司） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.AI

Comments Accepted at Human-Centered Robot Autonomy for Human-Robot Teams (HuRoboT) at IEEE RO-MAN 2025, Eindhoven, the Netherlands

Describe What You See with Multimodal Large Language Models to Enhance Video Recommendations

Marco De Nadai, Andreas Damianou, Mounia Lalmas

机构 * Spotify Denmark（Spotify丹麦分公司） ； Spotify United Kingdom（Spotify英国分公司） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV

DexVLA: Vision-Language Model with Plug-In Diffusion Expert for General Robot Control

Junjie Wen, Yichen Zhu, Jinming Li, Zhibin Tang, Chaomin Shen, Feifei Feng

专题命中 GUI与屏幕智能体 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV

Comments The webpage is at https://dex-vla.github.io/. DexVLA is accepted by CoRL 2025

UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis

Xinyi Liu, Xiaoyi Zhang, Ziyun Zhang, Yan Lu

专题命中 GUI与屏幕智能体 ：grounding(title,abstract)；vision-language model(abstract)；分类 cs.CV

WMNav: Integrating Vision-Language Models into World Models for Object Goal Navigation

Dujun Nie, Xianda Guo, Yiqun Duan, Ruijun Zhang, Long Chen

机构 * Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） ； School of Computer Science, Wuhan University（武汉大学计算机学院） ； School of Computer Science, University of Technology Sydney（悉尼大学计算机学院） ； IAIR, Xi’an Jiaotong University（西安交通大学IAIR） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV

Comments 8 pages, 5 figures

Journal ref IROS 2025

Self-Consistency in Vision-Language Models for Precision Agriculture: Multi-Response Consensus for Crop Disease Management

Mihir Gupta, Abhay Mangla, Ross Greer, Pratik Desai

机构 * The Harker School, USA（哈克尔学校） ； Dougherty Valley High School, USA（道格拉斯谷高中） ； Kissan.ai, USA（加州大学梅尔德分校） ； University of California, Merced, USA 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV

History-Augmented Vision-Language Models for Frontier-Based Zero-Shot Object Navigation

Mobin Habibpour, Fatemeh Afghah

机构 * Holcombe Department of Electrical and Computer Engineering, Clemson University, Clemson, SC, USA（霍尔科姆电气与计算机工程系，克莱姆森大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.AI

SAP-Bench: Benchmarking Multimodal Large Language Models in Surgical Action Planning

Mengya Xu, Zhongzhen Huang, Dillan Imans, Yiru Ye, Xiaofan Zhang, Qi Dou

专题命中 GUI与屏幕智能体 ：multimodal large language model(title,abstract)；MLLM(abstract)；分类 cs.CV

Comments The authors could not reach a consensus on the final version of this paper, necessitating its withdrawal

Mem2Ego: Empowering Vision-Language Models with Global-to-Ego Memory for Long-Horizon Embodied Navigation

Lingfeng Zhang, Yuecheng Liu, Zhanguang Zhang, Matin Aghaei, Yaochen Hu, Hongjian Gu, Mohammad Ali Alomrani, David Gamaliel Arcos Bravo, Raika Karimi, Atia Hamidizadeh, Haoping Xu, Guowei Huang, Zhanpeng Zhang, Tongtong Cao, Weichao Qiu, Xingyue Quan, Jianye Hao, Yuzheng Zhuang, Yingxue Zhang

专题命中 GUI与屏幕智能体 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.AI

RATE-Nav: Region-Aware Termination Enhancement for Zero-shot Object Navigation with Vision-Language Models

Junjie Li, Nan Zhang, Xiaoyang Qu, Kai Lu, Guokuan Li, Jiguang Wan, Jianzong Wang

机构 * Huazhong University of Science and Technology（华中科技大学） ； Ping An Technology (Shenzhen) Co., Ltd.（平安科技（深圳）有限公司） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(title)；visual language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Accepted by the 63rd Annual Meeting of the Association for Computational Linguistics (ACL 2025)

ScreenExplorer: Training a Vision-Language Model for Diverse Exploration in Open GUI World

Runliang Niu, Jinglong Ji, Yi Chang, Qi Wang

机构 * School of Artificial Intelligence, Jilin University（吉林大学人工智能学院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.AI

Co-NavGPT: Multi-Robot Cooperative Visual Semantic Navigation Using Vision Language Models

Bangguo Yu, Qihao Yuan, Kailai Li, Hamidreza Kasaei, Ming Cao

机构 * Faculty of Science and Engineering, University of Groningen（工程学院，格罗宁根大学） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.AI

Comments 8 pages, 4 figures

LiteWebAgent: The Open-Source Suite for VLM-Based Web-Agent Applications

Danqing Zhang, Balaji Rama, Jingyi Ni, Shiying He, Fu Zhao, Kunyu Chen, Arnold Chen, Junyu Cao

机构 * Rutgers University（罗格斯大学） ； The University of Texas at Austin（德克萨斯大学奥斯汀分校） 纠错

专题命中 GUI与屏幕智能体 ：VLM(title,abstract)；grounding(abstract)；分类 cs.AI

Agreeing to Interact in Human-Robot Interaction using Large Language Models and Vision Language Models

Kazuhiro Sasabuchi, Naoki Wake, Atsushi Kanehira, Jun Takamatsu, Katsushi Ikeuchi

专题命中 GUI与屏幕智能体 ：vision language model(title,abstract)；VLM(abstract)；分类 cs.LG

AppVLM: A Lightweight Vision Language Model for Online App Control

Georgios Papoudakis, Thomas Coste, Zhihao Wu, Jianye Hao, Jun Wang, Kun Shao

专题命中 GUI与屏幕智能体 ：vision language model(title)；vision-language model(abstract)；VLM(abstract)；分类 cs.AI

CogAgent: A Visual Language Model for GUI Agents

Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxuan Zhang, Juanzi Li, Bin Xu, Yuxiao Dong, Ming Ding, Jie Tang

专题命中 GUI与屏幕智能体 ：visual language model(title,abstract)；VLM(abstract)；分类 cs.CV

Comments CVPR 2024 (Highlight), 27 pages, 19 figures

Read Anywhere Pointed: Layout-aware GUI Screen Reading with Tree-of-Lens Grounding

Yue Fan, Lei Ding, Ching-Chen Kuo, Shan Jiang, Yang Zhao, Xinze Guan, Jie Yang, Yi Zhang, Xin Eric Wang

专题命中 GUI与屏幕智能体 ：grounding(title,abstract)；multimodal large language model(abstract)；分类 cs.CV

Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototyping

Ryan Li, Yanzhe Zhang, Diyi Yang

专题命中 GUI与屏幕智能体 ：vision-language model(title)；vision language model(abstract)；VLM(abstract)；分类 cs.AI

Comments preprint, 9 pages

BUMBLE: Unifying Reasoning and Acting with Vision-Language Models for Building-wide Mobile Manipulation

Rutav Shah, Albert Yu, Yifeng Zhu, Yuke Zhu, Roberto Martín-Martín

专题命中 GUI与屏幕智能体 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.AI

Comments 7 Figures, 2 Tables, 11 Pages

Affordance Perception by a Knowledge-Guided Vision-Language Model with Efficient Error Correction

Gertjan Burghouts, Marianne Schaaphok, Michael van Bekkum, Wouter Meijer, Fieke Hillerström, Jelle van Mil

专题命中 GUI与屏幕智能体 ：vision-language model(title,abstract)；VLM(abstract)；分类 cs.CV

Comments 15 pages

Journal ref International Conference on Pattern Recognition and Artificial Intelligence (ICPRAI) 2024

NaVid: Video-based VLM Plans the Next Step for Vision-and-Language Navigation

Jiazhao Zhang, Kunyu Wang, Rongtao Xu, Gengze Zhou, Yicong Hong, Xiaomeng Fang, Qi Wu, Zhizheng Zhang, He Wang

专题命中 GUI与屏幕智能体 ：VLM(title,abstract)；vision language model(abstract)；分类 cs.CV

Comments Accepted by Robotics: Science and Systems (RSS 2024)

Enhancing Video Transformers for Action Understanding with VLM-aided Training

Hui Lu, Hu Jian, Ronald Poppe, Albert Ali Salah

专题命中 GUI与屏幕智能体 ：VLM(title,abstract)；visual language model(abstract)；分类 cs.CV

Interactive Navigation in Environments with Traversable Obstacles Using Large Language and Vision-Language Models

Zhen Zhang, Anran Lin, Chun Wai Wong, Xiangyu Chu, Qi Dou, K. W. Samuel Au

专题命中 GUI与屏幕智能体 ：vision-language model(title,abstract)；grounding(abstract)；分类 cs.AI

Comments Accepted by 2024 IEEE International Conference on Robotics and Automation (ICRA), 7 pages, 8 figures

VTM-Nav: Harnessing Cross-Episode Experience for Object-Goal Navigation with Hierarchical Visual-Topological Memory

VTM-Nav：用于跨情节对象目标导航的分层视觉拓扑记忆

Xiaoran Xu, Yupeng Wu, Tianyu Xue, Yifan Xu, Xuanran Dong, Xiaoshan Yang, Changsheng Xu

机构 * MAIS, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所模式识别国家重点实验室） ； School of Advanced Interdisciplinary Sciences, University of Chinese Academy of Sciences（中国科学院大学交叉科学学院） ； Tsinghua University（清华大学） ； University of Chinese Academy of Sciences（中国科学院大学） 纠错

专题命中 GUI与屏幕智能体 ：VLM(summary_cn,abstract_cn)；vision-language model(abstract)；分类 cs.CV、cs.AI

AI总结 研究跨情节对象目标导航问题，提出无训练的VLM导航框架\method，其带有分层视觉拓扑记忆，通过粗到细匹配检索经验，在三个基准测试中性能最佳，证明跨数据集结构化视觉拓扑经验复用有效且鲁棒。