视觉大模型 / VLM

See-Control: A Multimodal Agent Framework for Smartphone Interaction with a Robotic Arm

See-Control: 一种多模态代理框架用于智能手机与机械臂的交互

Haoyu Zhao, Weizhong Ding, Yuhao Yang, Zheng Tian, Linyi Yang, Kun Shao, Jun Wang

机构 * University College London（伦敦大学学院） ； Imperial College London（伦敦帝国学院） ； Huawei Noah’s Ark Lab（华为诺亚实验室） ； ShanghaiTech University（上海科技大学） ； Southern University of Science（南方科技大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

AI总结 See-Control通过直接与机械臂交互实现智能手机操作，无需ADB或系统后台访问，提供了一个平台无关的多模态代理框架。

VideoCAD: A Dataset and Model for Learning Long-Horizon 3D CAD UI Interactions from Video

Brandon Man, Ghadi Nehme, Md Ferdous Alam, Faez Ahmed

机构 * Massachusetts Institute of Technology（麻省理工学院） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

OmniResponse: Online Multimodal Conversational Response Generation in Dyadic Interactions

Cheng Luo, Jianghui Wang, Bing Li, Siyang Song, Bernard Ghanem

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments 25 pages, 9 figures

Retrieval-augmented GUI Agents with Generative Guidelines

Ran Xu, Kaixin Ma, Wenhao Yu, Hongming Zhang, Joyce C. Ho, Carl Yang, Dong Yu

机构 * Emory University（埃默里大学） ； Tencent AI Lab（腾讯AI实验室） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI、cs.LG

Comments Accepted to EMNLP 2025 (Main Conference)

Human-like Navigation in a World Built for Humans

Bhargav Chandaka, Gloria X. Wang, Haozhe Chen, Henry Che, Albert J. Zhai, Shenlong Wang

机构 * University of Illinois Urbana-Champaign（伊利诺伊大学厄巴纳-香槟分校） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments CoRL 2025. Project website: https://reasonnav.github.io/

AgriVLN: Vision-and-Language Navigation for Agricultural Robots

Xiaobei Zhao, Xingqi Lyu, Xiang Li

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

A Survey on Vision-Language-Action Models for Autonomous Driving

Sicong Jiang, Zilin Huang, Kangan Qian, Ziang Luo, Tianze Zhu, Yang Zhong, Yihong Tang, Menglin Kong, Yunlong Wang, Siwen Jiao, Hao Ye, Zihao Sheng, Xin Zhao, Tuopu Wen, Zheng Fu, Sikai Chen, Kun Jiang, Diange Yang, Seongjin Choi, Lijun Sun

机构 * McGill University（麦吉尔大学） ； Tsinghua University（清华大学） ； Xiaomi Corporation（小米公司） ； University of Wisconsin–Madison（威斯康星大学麦迪逊分校） ； University of Minnesota–Twin Cities（明尼苏达大学双城分校） ； State Key Laboratory of Intelligent Green Vehicle and Mobility, Tsinghua University（智能绿色车辆与移动国家重点实验室，清华大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

An Open-Source Software Toolkit & Benchmark Suite for the Evaluation and Adaptation of Multimodal Action Models

Pranav Guruprasad, Yangyue Wang, Sudipta Chowdhury, Jaewoo Song, Harshvardhan Sikka

机构 * Metarch ； Manifold Research ； Georgia Institute of Technology（佐治亚理工学院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.LG

Comments ICML CodeML Workshop, 13 Pages, 6 Figures, 2 Tables

GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behavior

Penghao Wu, Shengnan Ma, Bo Wang, Jiaheng Yu, Lewei Lu, Ziwei Liu

机构 * S-Lab, Nanyang Technological University（南洋理工大学S实验室） ； SenseTime Research（商汤科技研究院） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments Project Page at https://penghao-wu.github.io/GUI_Reflection/

DiffVLA: Vision-Language Guided Diffusion Planning for Autonomous Driving

Anqing Jiang, Yu Gao, Zhigang Sun, Yiru Wang, Jijun Wang, Jinghao Chai, Qian Cao, Yuweng Heng, Hao Jiang, Yunda Dong, Zongzheng Zhang, Xianda Guo, Hao Sun, Hao Zhao

机构 * RIX, Bosch（博世 RIX） ； AIR, Tsinghua University（清华大学人工智能研究院） ； Shanghai University（上海大学） ； Shanghai Jiao Tong University（上海交通大学） ； Southeast University Team（东南大学团队） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments 4pages

HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation

Yi Li, Yuquan Deng, Jesse Zhang, Joel Jang, Marius Memmel, Raymond Yu, Caelan Reed Garrett, Fabio Ramos, Dieter Fox, Anqi Li, Abhishek Gupta, Ankit Goyal

机构 * NVIDIA ； University of Washington（华盛顿大学） ； University of Southern California（南加州大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments update related work and results on VQA benchmarks

NORA: A Small Open-Sourced Generalist Vision Language Action Model for Embodied Tasks

Chia-Yu Hung, Qi Sun, Pengfei Hong, Amir Zadeh, Chuan Li, U-Xuan Tan, Navonil Majumder, Soujanya Poria

机构 * Singapore University of Technology and Design（新加坡科技设计大学） ； Lambda Labs（Lambda实验室） 纠错

专题命中 GUI与屏幕智能体 ：visual reasoning(abstract)；grounding(abstract)；分类 cs.CV、cs.AI

Olympus: A Universal Task Router for Computer Vision Tasks

Yuanze Lin, Yunsheng Li, Dongdong Chen, Weijian Xu, Ronald Clark, Philip H. S. Torr

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted to CVPR 2025, Project webpage: http://yuanze-lin.me/Olympus_page/

GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding

Dongping Chen, Yue Huang, Siyuan Wu, Jingyu Tang, Liuyi Chen, Yilin Bai, Zhigang He, Chenlong Wang, Huichi Zhou, Yiqiang Li, Tianshuo Zhou, Yue Yu, Chujie Gao, Qihui Zhang, Yi Gui, Zhen Li, Yao Wan, Pan Zhou, Jianfeng Gao, Lichao Sun

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ICLR 2025

Tur[k]ingBench: A Challenge Benchmark for Web Agents

Kevin Xu, Yeganeh Kordi, Tanay Nayak, Adi Asija, Yizhong Wang, Kate Sanders, Adam Byerly, Jingyu Zhang, Benjamin Van Durme, Daniel Khashabi

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；InternVL(abstract)；分类 cs.CV、cs.AI

Iris: Breaking GUI Complexity with Adaptive Focus and Self-Refining

Zhiqi Ge, Juncheng Li, Xinglei Pang, Minghe Gao, Kaihang Pan, Wang Lin, Hao Fei, Wenqiao Zhang, Siliang Tang, Yueting Zhuang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

HiRT: Enhancing Robotic Control with Hierarchical Robot Transformers

Jianke Zhang, Yanjiang Guo, Xiaoyu Chen, Yen-Jen Wang, Yucheng Hu, Chengming Shi, Jianyu Chen

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted to CORL 2024

Sharingan: Extract User Action Sequence from Desktop Recordings

Yanting Chen, Yi Ren, Xiaoting Qin, Jue Zhang, Kehong Yuan, Lu Han, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

ClickAgent: Enhancing UI Location Capabilities of Autonomous Agents

Jakub Hoscilowicz, Bartosz Maj, Bartosz Kozakiewicz, Oleksii Tymoshchuk, Artur Janicki

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI、cs.LG

Comments The code for ClickAgent is available at github.com/Samsung/ClickAgent

EAGLE: Egocentric AGgregated Language-video Engine

Jing Bi, Yunlong Tang, Luchuan Song, Ali Vosoughi, Nguyen Nguyen, Chenliang Xu

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ACMMM 24

Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset

Hugo Laurençon, Léo Tronchon, Victor Sanh

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

GPT-4V(ision) is a Generalist Web Agent, if Grounded

Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su

专题命中 GUI与屏幕智能体 ：visual question answering(abstract)；grounding(abstract)；分类 cs.CV、cs.AI

ILuvUI: Instruction-tuned LangUage-Vision modeling of UIs from Machine Conversations

Yue Jiang, Eldon Schoop, Amanda Swearngin, Jeffrey Nichols

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV、cs.AI

RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control

Anthony Brohan, Noah Brown, Justice Carbajal, Yevgen Chebotar, Xi Chen, Krzysztof Choromanski, Tianli Ding, Danny Driess, Avinava Dubey, Chelsea Finn, Pete Florence, Chuyuan Fu, Montse Gonzalez Arenas, Keerthana Gopalakrishnan, Kehang Han, Karol Hausman, Alexander Herzog, Jasmine Hsu, Brian Ichter, Alex Irpan, Nikhil Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Isabel Leal, Lisa Lee, Tsang-Wei Edward Lee, Sergey Levine, Yao Lu, Henryk Michalewski, Igor Mordatch, Karl Pertsch, Kanishka Rao, Krista Reymann, Michael Ryoo, Grecia Salazar, Pannag Sanketi, Pierre Sermanet, Jaspiar Singh, Anikait Singh, Radu Soricut, Huong Tran, Vincent Vanhoucke, Quan Vuong, Ayzaan Wahid, Stefan Welker, Paul Wohlhart, Jialin Wu, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Tianhe Yu, Brianna Zitkovich

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV、cs.LG

Comments Website: https://robotics-transformer.github.io/

CLIP-Nav: Using CLIP for Zero-Shot Vision-and-Language Navigation

Vishnu Sashank Dorbala, Gunnar Sigurdsson, Robinson Piramuthu, Jesse Thomason, Gaurav S. Sukhatme

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)；分类 cs.CV、cs.AI

Comments 8 pages, Accepted at LangRob Workshop at Conference on Robot Learning (CoRL), 2022

Online Robot Navigation and Manipulation with Distilled Vision-Language Models

Kangcheng Liu

专题命中 GUI与屏幕智能体 ：vision-language model(title)

Comments International Conference on Robotics and Automation (ICRA)

Grounding Robot Navigation in Self-Defense Law

James Zhu, Anoushka Shrivastava, Aaron M. Johnson

专题命中 GUI与屏幕智能体 ：grounding(title)

Comments To Appear in RO-MAN 2023

Act2Intention: A Benchmark For Developing Active Mobile Agents Through Inferring User Intention from GUI Actions

Act2Intention：通过从GUI操作推断用户意图开发主动移动智能体的基准

Xiaokai Yan, Jingtao Ding, Yong Li, Zhiwen Yu

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract_cn)；分类 cs.AI

AI总结 本文提出Act2Intention框架及基准数据集，构建主动移动智能体，经实验验证该基准可显著提升智能体意图理解、预测与执行性能，为主动智能体研究提供标准化平台。

Lost in Reconstruction: Aligning Action Representations with Language in Vision-Language-Action Models

在重建中迷失：将视觉-语言-动作模型中的动作表示与语言对齐

Li Wenjie, Yash Jangir, Ignacy Stepka, Yash Agarwal, Marion Kipsang, Yonatan Bisk

机构 * Carnegie Mellon University（卡内基梅隆大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)；分类 cs.AI

AI总结 该研究针对视觉-语言-动作模型中动作表示与语言脱节的问题，提出SALT语义对齐动作标记器，在SimplerEnv中使策略平均成功率达71.9%，显著优于基线方法。

NormAct: Benchmarking Embodied Agents' Proactive Compliance with Unspoken Social Norms

NormAct：具身规划中隐藏社会规范遵守的基准

Shiyun Zhao, Xinwei Song, Tianyu Guo, Xiaomeng Gao, Mingyuan Liu, Xu Han, Yuanyuan Zhang, Zhenliang Zhang, Xue Feng, Bo Dai

机构 * State Key Laboratory of General Artificial Intelligence, Beijing Institute for General Artificial Intelligence (BIGAI)（通用人工智能国家重点实验室，北京通用人工智能研究院） ； China Academy of Information and Communications Technology（中国信息通信研究院） ； ShanghaiTech University（上海科技大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI

AI总结 提出NormAct基准，评估多模态大语言模型在具身规划中遵守隐藏社会规范的能力，发现模型在67.3%情况下达成显式目标，但仅26.4%遵守隐藏规范；提出NormPerceptor方法将任务成功率从24.2%提升至46.7%。

Comments This version revises the paper content and adds substantial details on the benchmark design and experimental setup