视觉大模型 / VLM

Generalist Robot Manipulation beyond Action Labeled Data

Alexander Spiridonov, Jan-Nico Zaech, Nikolay Nikolov, Luc Van Gool, Danda Pani Paudel

机构 * ETH Zurich, Switzerland（苏黎世联邦理工学院，瑞士） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments Accepted at Conference on Robot Learning 2025

SINGER: An Onboard Generalist Vision-Language Navigation Policy for Drones

Maximilian Adang, JunEn Low, Ola Shorinwa, Mac Schwager

机构 * Department of Aeronautics and Astronautics, Stanford University（航空与航天系，斯坦福大学） ； Department of Mechanical Engineering, Stanford University（机械工程系，斯坦福大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Embodied Navigation Foundation Model

Jiazhao Zhang, Anqi Li, Yunpeng Qi, Minghan Li, Jiahang Liu, Shaoan Wang, Haoran Liu, Gengze Zhou, Yuze Wu, Xingxing Li, Yuxin Fan, Wenjun Li, Zhibo Chen, Fei Gao, Qi Wu, Zhizheng Zhang, He Wang

机构 * Peking University（北京大学） ； Galbot ； USTC（中国科学技术大学） ； BAAI（百度人工智能研究院） ； University of Adelaide（阿德莱德大学） ； Zhejiang University（浙江大学） ； Differential Robotics Project（差分机器人项目） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments Project Page: https://pku-epic.github.io/NavFoM-Web/

Sign Language: Towards Sign Understanding for Robot Autonomy

Ayush Agrawal, Joel Loo, Nicky Zimmerman, David Hsu

机构 * Smart Systems Institute, National University of Singapore（新加坡国立大学智能系统研究所） ； Northeastern University（东北大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments This work has been submitted to the IEEE for possible publication

SpecifyUI: Supporting Iterative UI Design Intent Expression through Structured Specifications and Generative AI

Yunnong Chen, Chengwei Shi, Liuqing Chen

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments 27 pages, 12 figures

Enhancing Multi-Robot Semantic Navigation Through Multimodal Chain-of-Thought Score Collaboration

Zhixuan Shen, Haonan Luo, Kexun Chen, Fengmao Lv, Tianrui Li

机构 * Zhixuan Shen, Haonan Luo, Kexun Chen, Fengmao Lv, Tianrui Li（作者） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(abstract)

Comments 16 pages, 10 figures, Extended Version of accepted AAAI 2025 Paper

AI-Supported Mini-Labs: Combining Smartphone-Based Experiments and Multimodal AI

Jochen Kuhn, David J. Rakestraw, Stefan Küchemann, Patrik Vogt

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

MVISU-Bench: Benchmarking Mobile Agents for Real-World Tasks by Multi-App, Vague, Interactive, Single-App and Unethical Instructions

Zeyu Huang, Juyuan Wang, Longfeng Chen, Boyi Xiao, Leng Cai, Yawen Zeng, Jin Xu

机构 * South China University of Technology（华南理工大学） ； Pazhou Lab（Pazhou 实验室） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(abstract)

Comments ACM MM 2025

Real-Time 3D Vision-Language Embedding Mapping

Christian Rauch, Björn Ellensohn, Linus Nwankwo, Vedant Dave, Elmar Rueckert

机构 * Technical University of Leoben（莱博恩技术大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

GUI-ReRank: Enhancing GUI Retrieval with Multi-Modal LLM-based Reranking

Kristian Kolthoff, Felix Kretzer, Christian Bartelt, Alexander Maedche, Simone Paolo Ponzetto

专题命中 GUI与屏幕智能体 ：MLLM(abstract)

CookBench: A Long-Horizon Embodied Planning Benchmark for Complex Cooking Scenarios

Muzhen Cai, Xiubo Chen, Yining An, Jiaxin Zhang, Xuesong Wang, Wang Xu, Weinan Zhang, Ting Liu

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments 9 pages, 5 figures

SA-GCS: Semantic-Aware Gaussian Curriculum Scheduling for UAV Vision-Language Navigation

Hengxing Cai, Jinhan Dong, Yijie Rao, Jingcheng Deng, Jingjun Tan, Qien Chen, Haidong Wang, Zhen Wang, Shiyu Huang, Agachai Sumalee, Renxin Zhong

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

FMimic: Foundation Models are Fine-grained Action Learners from Human Videos

Guangyan Chen, Meiling Wang, Te Cui, Yao Mu, Haoyang Lu, Zicai Peng, Mengxiao Hu, Tianxing Zhou, Mengyin Fu, Yi Yang, Yufeng Yue

机构 * Beijing Institute of Technology, Beijing, P. R. China（北京理工大学） ； The University of Hong Kong, Hong Kong, P. R. China（香港大学） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(abstract)

Comments accepted to International Journal of Robotics Research(IJRR)

MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation

Ning Li, Xiangmou Qu, Jiamu Zhou, Jun Wang, Muning Wen, Kounianhua Du, Xingyu Lou, Qiuying Peng, Jun Wang, Weinan Zhang

机构 * Shanghai Jiao Tong University（上海交通大学） ； OPPO Research Institute（OPPO研究院） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments A technical report on a GUI agent based on multi-agent systems

VL-Explore: Zero-shot Vision-Language Exploration and Target Discovery by Mobile Robots

Yuxuan Zhang, Adnan Abdullah, Sanjeev J. Koppal, Md Jahidul Islam

机构 * RoboPI Laboratory, Dept. of ECE, University of Florida (UF)（罗波实验室，电子工程系，佛罗里达大学） ； FOCUS Laboratory, Dept. of ECE, University of Florida (UF)（焦点实验室，电子工程系，佛罗里达大学） ； Amazon Robotics（亚马逊机器人技术） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments V2, includes suppl as appendix

EdgeVLA: Efficient Vision-Language-Action Models

Paweł Budzianowski, Wesley Maa, Matthew Freed, Jingxiang Mo, Winston Hsiao, Aaron Xie, Tomasz Młoduchowski, Viraj Tipnis, Benjamin Bolte

机构 * K-Scale Labs（K-Scale实验室） ； McGill University（麦吉尔大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Journal ref IROS-MoMA3 Workshop 2024

MOSU: Autonomous Long-range Robot Navigation with Multi-modal Scene Understanding

Jing Liang, Kasun Weerakoon, Daeun Song, Senthurbavan Kirubaharan, Xuesu Xiao, Dinesh Manocha

机构 * University of Maryland, College Park MD, 20740, USA（马里兰大学） ； Goerge Mason University, Fairfax, VA, 22030, USA（乔治·马歇尔大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Narrate2Nav: Real-Time Visual Navigation with Implicit Language Reasoning in Human-Centric Environments

Amirreza Payandeh, Anuj Pokhrel, Daeun Song, Marcos Zampieri, Xuesu Xiao

机构 * Department of Computer Science, George Mason University（计算机科学系，乔治·马歇尔大学） ； Department of Information Sciences and Technology, George Mason University（信息科学与技术系，乔治·马歇尔大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

DesignCoder: Hierarchy-Aware and Self-Correcting UI Code Generation with Large Language Models

Yunnong Chen, Shixian Ding, YingYing Zhang, Wenkai Chen, Jinzhou Du, Lingyun Sun, Liuqing Chen

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments 11 pages,6 figures

RationalVLA: A Rational Vision-Language-Action Model with Dual System

Wenxuan Song, Jiayi Chen, Wenxue Li, Xu He, Han Zhao, Can Cui, Pengxiang Ding Shiyan Su, Feilong Tang, Xuelian Cheng, Donglin Wang, Zongyuan Ge, Xinhu Zheng, Zhe Liu, Hesheng Wang, Haoang Li

机构 * The Hong Kong University of Science and Technology (Guangzhou)（香港科技大学（广州）） ； Westlake University（西湖大学） ； Monash University（墨尔本大学） ； Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments 14 pages

GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs

Ruofan Liu, Xiwen Teoh, Yun Lin, Guanjie Chen, Ruofei Ren, Denys Poshyvanyk, Jin Song Dong

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Bridging Perception and Action: Spatially-Grounded Mid-Level Representations for Robot Generalization

Jonathan Yang, Chuyuan Kelly Fu, Dhruv Shah, Dorsa Sadigh, Fei Xia, Tingnan Zhang

机构 * Stanford University（斯坦福大学） ； Google DeepMind（谷歌DeepMind） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)

Comments 16 pages, 13 figures

FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents

Bobo Li, Yuheng Wang, Hao Fei, Juncheng Li, Wei Ji, Mong-Li Lee, Wynne Hsu

机构 * National University of Singapore（新加坡国立大学） ； Wuhan University（武汉大学） ； Zhejiang University（浙江大学） ； Nanjing University（南京大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments 8 pages, 7 figures

UICopilot: Automating UI Synthesis via Hierarchical Code Generation from Webpage Designs

Yi Gui, Zhen Li, Zhongyi Zhang, Yao Wan, Dongping Chen, Hongyu Zhang, Yi Su, Bohua Chen, Xing Zhou, Wenbin Jiang, Xiangliang Zhang

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments WWW' 2025

CLTP: Contrastive Language-Tactile Pre-training for 3D Contact Geometry Understanding

Wenxuan Ma, Xiaoge Cao, Yixiang Zhang, Chaofan Zhang, Shaobo Yang, Peng Hao, Bin Fang, Yinghao Cai, Shaowei Cui, Shuo Wang

机构 * Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） ； Beihang University（北航） ； Beijing University of Posts and Telecommunications（北京邮电大学） ； Samsung Research China（三星中国研究院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

Comments 16 pages

LogisticsVLN: Vision-Language Navigation For Low-Altitude Terminal Delivery Based on Agentic UAVs

Xinyuan Zhang, Yonglin Tian, Fei Lin, Yue Liu, Jing Ma, Kornélia Sára Szatmáry, Fei-Yue Wang

机构 * School of Artificial Intelligence, University of Chinese Academy of Sciences（中国科学院大学人工智能学院） ； State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所多模态人工智能系统国家重点实验室） ； Department of Engineering Science, Faculty of Innovation Engineering, Macau University of Science and Technology（澳门科技大学创新工程学院工程科学系） ； China Ship Research and Development Academy（中国船舶科研 Academy） ； Obuda University（奥布达大学） ； State Key Laboratory for Management and Control of Complex Systems, Chinese Academy of Sciences（中国科学院复杂系统管理与控制国家重点实验室） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction

Yiheng Xu, Zekun Wang, Junli Wang, Dunjie Lu, Tianbao Xie, Amrita Saha, Doyen Sahoo, Tao Yu, Caiming Xiong

专题命中 GUI与屏幕智能体 ：grounding(abstract)

Comments ICML 2025

Media Content Atlas: A Pipeline to Explore and Investigate Multidimensional Media Space using Multimodal LLMs

Merve Cerit, Eric Zelikman, Mu-Jung Cho, Thomas N. Robinson, Byron Reeves, Nilam Ram, Nick Haber

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments Accepted to CHI 2025, in press. See the project page at mediacontentatlas.github.io

Early Accessibility: Automating Alt-Text Generation for UI Icons During App Development

Sabrina Haque, Christoph Csallner

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)

ScreenLLM: Stateful Screen Schema for Efficient Action Understanding and Prediction

Yiqiao Jin, Stefano Petrangeli, Yu Shen, Gang Wu

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)

Comments Accepted to MM4SG Workshop at The Web Conference 2025