视觉大模型 / VLM

HybridVLA: Collaborative Diffusion and Autoregression in a Unified Vision-Language-Action Model

Jiaming Liu, Hao Chen, Pengju An, Zhuoyang Liu, Renrui Zhang, Chenyang Gu, Xiaoqi Li, Ziyu Guo, Sixiang Chen, Mengzhen Liu, Chengkai Hou, Mengdi Zhao, KC alex Zhou, Pheng-Ann Heng, Shanghang Zhang

机构 * State Key Laboratory of Multimedia Information Processing, School of Computer Science, Peking University（多媒体信息处理国家重点实验室，计算机学院，北京大学） ； Beijing Academy of Artificial Intelligence (BAAI)（北京人工智能研究院） ； CUHK（香港中文大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

From Intention to Execution: Probing the Generalization Boundaries of Vision-Language-Action Models

Irving Fang, Juexiao Zhang, Shengbang Tong, Chen Feng

机构 * New York University（纽约大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Under review

Surfer-H Meets Holo1: Cost-Efficient Web Agent Powered by Open Weights

Mathieu Andreux, Breno Baldas Skuk, Hamza Benchekroun, Emilien Biré, Antoine Bonnet, Riaz Bordie, Nathan Bout, Matthias Brunel, Pierre-Louis Cedoz, Antoine Chassang, Mickaël Chen, Alexandra D. Constantinou, Antoine d'Andigné, Hubert de La Jonquière, Aurélien Delfosse, Ludovic Denoyer, Alexis Deprez, Augustin Derupti, Michael Eickenberg, Mathïs Federico, Charles Kantor, Xavier Koegler, Yann Labbé, Matthew C. H. Lee, Erwan Le Jumeau de Kergaradec, Amir Mahla, Avshalom Manevich, Adrien Maret, Charles Masson, Rafaël Maurin, Arturo Mena, Philippe Modard, Axel Moyal, Axel Nguyen Kerbel, Julien Revelle, Mats L. Richter, María Santos, Laurent Sifre, Maxime Theillard, Marc Thibault, Louis Thiry, Léo Tronchon, Nicolas Usunier, Tony Wu

机构 * H Company（H公司） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

Comments Alphabetical order

LoHoVLA: A Unified Vision-Language-Action Model for Long-Horizon Embodied Tasks

Yi Yang, Jiaxuan Sun, Siqi Kou, Yihan Wang, Zhijie Deng

机构 * Fudan University（复旦大学） ； ShanghaiTech University（上海科技大学） ； Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；VLM(abstract)；分类 cs.AI

Knowledge Insulating Vision-Language-Action Models: Train Fast, Run Fast, Generalize Better

Danny Driess, Jost Tobias Springenberg, Brian Ichter, Lili Yu, Adrian Li-Bell, Karl Pertsch, Allen Z. Ren, Homer Walke, Quan Vuong, Lucy Xiaoyang Shi, Sergey Levine

机构 * Physical Intelligence 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.LG

AdInject: Real-World Black-Box Attacks on Web Agents via Advertising Delivery

Haowei Wang, Junjie Wang, Xiaojun Jia, Rupeng Zhang, Mingyang Li, Zhe Liu, Yang Liu, Qing Wang

机构 * State Key Laboratory of Intelligent Game（智能游戏国家重点实验室） ； Institute of Software, Chinese Academy of Sciences（中国科学院软件研究所） ； University of Chinese Academy of Sciences（中国科学院大学） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

FM-Planner: Foundation Model Guided Path Planning for Autonomous Drone Navigation

Jiaping Xiao, Cheng Wen Tsao, Yuhang Zhang, Mir Feroskhan

机构 * School of Mechanical and Aerospace Engineering, Nanyang Technological University（机械与航空航天工程学院，南洋理工大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

Comments This work has been submitted for possible publication

QUART-Online: Latency-Free Large Multimodal Language Model for Quadruped Robot Learning

Xinyang Tong, Pengxiang Ding, Yiguo Fan, Donglin Wang, Wenjie Zhang, Can Cui, Mingyang Sun, Han Zhao, Hongyin Zhang, Yonghao Dang, Siteng Huang, Shangke Lyu

机构 * MiLAB, Westlake University, Hangzhou, 310030, China（西交利物浦大学微实验室，杭州，310030，中国） ； Zhejiang University, Hangzhou, 310027, China（浙江大学，杭州，310027，中国） ； Beijing University of Posts and Telecommunications, Beijing, 100876, China（北京邮电大学，北京，100876，中国） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Comments Accepted to ICRA 2025; Github page: https://quart-online.github.io

UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning

Zhengxi Lu, Yuxiang Chai, Yaxuan Guo, Xi Yin, Liang Liu, Hao Wang, Han Xiao, Shuai Ren, Guanjing Xiong, Hongsheng Li

机构 * vivo AI Lab（vivo人工智能实验室） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；multimodal large language model(abstract)；分类 cs.AI

Comments Updated UI-R1-E-3B

TinyClick: Single-Turn Agent for Empowering GUI Automation

Pawel Pawlowski, Krystian Zawistowski, Wojciech Lapacz, Adam Wiacek, Marcin Skorupa, Sebastien Postansque, Jakub Hoscilowicz

机构 * Warsaw University of Technology（华沙技术大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；MLLM(abstract)；分类 cs.AI

Comments Accepted to Interspeech 2025

Building a Stable Planner: An Extended Finite State Machine Based Planning Module for Mobile GUI Agent

Fanglin Mo, Junzhe Chen, Haoxuan Zhu, Xuming Hu

机构 * Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）） ； School of Computer Science & Engineering, South China University of Technology（华南理工大学计算机科学与工程学院） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；VLM(abstract)；分类 cs.AI

Comments 10 pages. Submitted to EMNLP 2025

Vision and Intention Boost Large Language Model in Long-Term Action Anticipation

Congqi Cao, Lanshu Hu, Yating Yu, Yanning Zhang

机构 * Northwestern Polytechnical University（西北工业大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Automatically Generating UI Code from Screenshot: A Divide-and-Conquer-Based Approach

Yuxuan Wan, Chaozheng Wang, Yi Dong, Wenxuan Wang, Shuqing Li, Yintong Huo, Michael R. Lyu

机构 * The Chinese University of Hong Kong（香港中文大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI

Comments Accepted by FSE 2025

SpiritSight Agent: Advanced GUI Agent with One Look

Zhiyuan Huang, Ziming Cheng, Junting Pan, Zhaohui Hou, Mingjie Zhan

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；grounding(abstract)；分类 cs.CV

Comments Paper accepted to CVPR 2025

SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation

Jingxuan Chen, Derek Yuen, Bin Xie, Yuhao Yang, Gongwei Chen, Zhihao Wu, Li Yixing, Xurui Zhou, Weiwen Liu, Shuai Wang, Kaiwen Zhou, Rui Shao, Liqiang Nie, Yasheng Wang, Jianye Hao, Jun Wang, Kun Shao

专题命中 GUI与屏幕智能体 ：grounding(abstract)；MLLM(abstract)；分类 cs.AI

Comments ICLR 2025 Spotlight

Does Chain-of-Thought Reasoning Help Mobile GUI Agent? An Empirical Study

Li Zhang, Longxi Gao, Mengwei Xu

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning

Siyin Wang, Zhaoye Fei, Qinyuan Cheng, Shiduo Zhang, Panpan Cai, Jinlan Fu, Xipeng Qiu

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV

TLA: Tactile-Language-Action Model for Contact-Rich Manipulation

Peng Hao, Chaofan Zhang, Dingzhe Li, Xiaoge Cao, Xiaoshuai Hao, Shaowei Cui, Shuo Wang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)；分类 cs.CV

Optimus-2: Multimodal Minecraft Agent with Goal-Observation-Action Conditioned Policy

Zaijing Li, Yuquan Xie, Rui Shao, Gongwei Chen, Dongmei Jiang, Liqiang Nie

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI

Comments Accept to CVPR 2025, Project page: https://cybertronagent.github.io/Optimus-2.github.io/

GenieBlue: Integrating both Linguistic and Multimodal Capabilities for Large Language Models on Mobile Devices

Xudong Lu, Yinghao Chen, Renshou Wu, Haohao Gao, Xi Chen, Xue Yang, Xiangyu Zhao, Aojun Zhou, Fangyuan Li, Yafei Wen, Xiaoxin Chen, Shuai Ren, Hongsheng Li

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Comments 14 pages

CHOP: Mobile Operating Assistant with Constrained High-frequency Optimized Subtask Planning

Yuqi Zhou, Shuai Wang, Sunhao Dai, Qinglin Jia, Zhaocheng Du, Zhenhua Dong, Jun Xu

专题命中 GUI与屏幕智能体 ：VLM(abstract)；visual language model(abstract)；分类 cs.AI

Lightweight Neural App Control

Filippos Christianos, Georgios Papoudakis, Thomas Coste, Jianye Hao, Jun Wang, Kun Shao

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

Comments ICLR 2025 (spotlight)

SKI Models: Skeleton Induced Vision-Language Embeddings for Understanding Activities of Daily Living

Arkaprava Sinha, Dominick Reilly, Francois Bremond, Pu Wang, Srijan Das

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；vision language model(abstract)；分类 cs.CV

On AI-Inspired UI-Design

Jialiang Wei, Anne-Lise Courbis, Thomas Lambolais, Gérard Dray, Walid Maalej

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

ACE: Action Concept Enhancement of Video-Language Models in Procedural Videos

Reza Ghoddoosian, Nakul Agarwal, Isht Dwivedi, Behzad Darisuh

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Accepted at WACV 2025

OS-ATLAS: A Foundation Action Model for Generalist GUI Agents

Zhiyong Wu, Zhenyu Wu, Fangzhi Xu, Yian Wang, Qiushi Sun, Chengyou Jia, Kanzhi Cheng, Zichen Ding, Liheng Chen, Paul Pu Liang, Yu Qiao

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)；分类 cs.CV

OccLLaMA: An Occupancy-Language-Action Generative World Model for Autonomous Driving

Julong Wei, Shanshuai Yuan, Pengfei Li, Qingda Hu, Zhongxue Gan, Wenchao Ding

专题命中 GUI与屏幕智能体 ：visual question answering(abstract)；MLLM(abstract)；分类 cs.CV

Spider2-V: How Far Are Multimodal Agents From Automating Data Science and Engineering Workflows?

Ruisheng Cao, Fangyu Lei, Haoyuan Wu, Jixuan Chen, Yeqiao Fu, Hongcheng Gao, Xinzhuang Xiong, Hanchong Zhang, Yuchen Mao, Wenjing Hu, Tianbao Xie, Hongshen Xu, Danyang Zhang, Sida Wang, Ruoxi Sun, Pengcheng Yin, Caiming Xiong, Ansong Ni, Qian Liu, Victor Zhong, Lu Chen, Kai Yu, Tao Yu

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；VLM(abstract)；分类 cs.AI

Comments 34 pages, 14 figures, 10 tables

Mobility VLA: Multimodal Instruction Navigation with Long-Context VLMs and Topological Graphs

Hao-Tien Lewis Chiang, Zhuo Xu, Zipeng Fu, Mithun George Jacob, Tingnan Zhang, Tsang-Wei Edward Lee, Wenhao Yu, Connor Schenck, David Rendleman, Dhruv Shah, Fei Xia, Jasmine Hsu, Jonathan Hoech, Pete Florence, Sean Kirmani, Sumeet Singh, Vikas Sindhwani, Carolina Parada, Chelsea Finn, Peng Xu, Sergey Levine, Jie Tan

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；VLM(abstract)；分类 cs.AI

E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion

Ke Wang, Tianyu Xia, Zhangxuan Gu, Yi Zhao, Shuheng Shen, Changhua Meng, Weiqiang Wang, Ke Xu

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Comments 9 pages, 5 figures, Under review