视觉大模型 / VLM

Impromptu VLA: Open Weights and Open Data for Driving Vision-Language-Action Models

Haohan Chi, Huan-ang Gao, Ziming Liu, Jianing Liu, Chenyu Liu, Jinwei Li, Kaisen Yang, Yangcheng Yu, Zeda Wang, Wenyi Li, Leichen Wang, Xingtao Hu, Hao Sun, Hang Zhao, Hao Zhao

机构 * AIR, Tsinghua University（清华大學人工智能研究院） ； Bosch Research（博世研究） ； IIIS, Tsinghua University（清华大學人工智能研究院） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV

Comments Project page: https://github.com/ahydchh/Impromptu-VLA

DynaMem: Online Dynamic Spatio-Semantic Memory for Open World Mobile Manipulation

Peiqi Liu, Zhanqiu Guo, Mohit Warke, Soumith Chintala, Chris Paxton, Nur Muhammad Mahi Shafiullah, Lerrel Pinto

机构 * New York University（纽约大学） ； Meta Inc.（Meta公司） ； Hello Robot Inc.（Hello Robot公司） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.LG

Comments Website: https://dynamem.github.io

VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning

Guanxing Lu, Wenkai Guo, Chubin Zhang, Yuheng Zhou, Haonan Jiang, Zifeng Gao, Yansong Tang, Ziwei Wang

机构 * Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院，清华大学） ； School of Electrical and Electronic Engineering, Nanyang Technological University（南洋理工大学电子与电气工程学院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

AgentOccam: A Simple Yet Strong Baseline for LLM-Based Web Agents

Ke Yang, Yao Liu, Sapana Chaudhary, Rasool Fakoor, Pratik Chaudhari, George Karypis, Huzefa Rangwala

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

DongbaMIE: A Multimodal Information Extraction Dataset for Evaluating Semantic Understanding of Dongba Pictograms

Xiaojun Bi, Shuo Li, Junyao Xing, Ziyue Wang, Fuwen Luo, Weizheng Qiao, Lu Han, Ziwei Sun, Peng Li, Yang Liu

机构 * College of Information and Engineering, Minzu University of China（中国民族大学信息与工程学院） ； Key Laboratory of Ethnic Language Intelligent Analysis and Security Governance of MOE, Minzu University of China（教育部民族语言智能分析与安全治理重点实验室） ； College of Information and Communication Engineering, Harbin Engineering University（哈尔滨工程大学信息与通信工程学院） ； Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University（清华大学人工智能研究院计算机科学与技术系） ； Institute for AI Industry Research (AIR), Tsinghua University（清华大学人工智能产业研究院） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV

Comments Our dataset can be obtained from: https://github.com/thinklis/DongbaMIE

MobileA3gent: Training Mobile GUI Agents Using Decentralized Self-Sourced Data from Diverse Users

Wenhao Wang, Mengying Yuan, Zijie Yu, Guangyi Liu, Rui Ye, Tian Jin, Siheng Chen, Yanfeng Wang

机构 * Zhejiang University（浙江大学） ； Shanghai Jiao Tong University（上海交通大学） ； Shanghai AI Laboratory（上海人工智能实验室） ； Multi-Agent Governance & Intelligence Crew (MAGIC)（多智能体治理与智能团队（MAGIC）） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.AI

AXIS: Efficient Human-Agent-Computer Interaction with API-First LLM-Based Agents

Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

机构 * Peking University（北京大学） ； Nanjing University（南京大学） ； Microsoft（微软） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI

SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model

Delin Qu, Haoming Song, Qizhi Chen, Yuanqi Yao, Xinyi Ye, Yan Ding, Zhigang Wang, JiaYuan Gu, Bin Zhao, Dong Wang, Xuelong Li

机构 * Shanghai AI Laboratory（上海人工智能实验室） ； Fudan University（复旦大学） ； Shanghai Jiao Tong University（上海交通大学） ； Zhejiang University（浙江大学） ； ShanghaiTech University（上海科技大学） ； Northwestern Polytechnical University（西北工业大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

Journal ref Robotics: Science and Systems, 2025

Flex: End-to-End Text-Instructed Visual Navigation from Foundation Model Features

Makram Chahine, Alex Quach, Alaa Maalouf, Tsun-Hsuan Wang, Daniela Rus

机构 * CSAIL, MIT（MIT计算机科学与人工智能实验室） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.AI

PADriver: Towards Personalized Autonomous Driving

Genghua Kou, Fan Jia, Weixin Mao, Yingfei Liu, Yucheng Zhao, Ziheng Zhang, Osamu Yoshie, Tiancai Wang, Ying Li, Xiangyu Zhang

机构 * Beijing Institute of Technology（北京理工大学） ； Megvii Technology（商汤科技） ； Waseda University（早稻田大学） 纠错

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.CV

Zero-shot Object-Centric Instruction Following: Integrating Foundation Models with Traditional Navigation

Sonia Raychaudhuri, Duy Ta, Katrina Ashton, Angel X. Chang, Jiuguang Wang, Bernadette Bucher

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

ScaleTrack: Scaling and back-tracking Automated GUI Agents

Jing Huang, Zhixiong Zeng, Wenkang Han, Yufeng Zhong, Liming Zheng, Shuai Fu, Jingyuan Chen, Lin Ma

机构 * Meituan（美团） ； Zhejiang University（浙江大学） ； University of Adelaide（阿德莱德大学） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

UFO2: The Desktop AgentOS

Chaoyun Zhang, He Huang, Chiming Ni, Jian Mu, Si Qin, Shilin He, Lu Wang, Fangkai Yang, Pu Zhao, Chao Du, Liqun Li, Yu Kang, Zhao Jiang, Suzhen Zheng, Rujia Wang, Jiaxu Qian, Minghua Ma, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang

机构 * Microsoft（微软公司） ； ZJU-UIUC Institute（浙大-伊利诺伊大学联合研究所） ； Nanjing University（南京大学） ； Peking University（北京大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI

Comments The source code of UFO2 is publicly available at https://github.com/microsoft/UFO/, with comprehensive documentation provided at https://microsoft.github.io/UFO/

Cracking the Code of Action: a Generative Approach to Affordances for Reinforcement Learning

Lynn Cherif, Flemming Kondrup, David Venuto, Ankit Anand, Doina Precup, Khimya Khetarpal

机构 * McGill University（麦吉尔大学） ； Mila ； Google DeepMind（谷歌DeepMind） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

MoLe-VLA: Dynamic Layer-skipping Vision Language Action Model via Mixture-of-Layers for Efficient Robot Manipulation

Rongyu Zhang, Menghang Dong, Yuan Zhang, Liang Heng, Xiaowei Chi, Gaole Dai, Li Du, Yuan Du, Shanghang Zhang

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI

Is Temporal Prompting All We Need For Limited Labeled Action Recognition?

Shreyank N Gowda, Boyan Gao, Xiao Gu, Xiaobo Jin

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Accepted in CVPR-W 2025

Seeing is Believing? Enhancing Vision-Language Navigation using Visual Perturbations

Xuesong Zhang, Jia Li, Yunbo Xu, Zhenzhen Hu, Richang Hong

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments 8 pages, 5 figures, accepted at IJCNN 2025

Multi-modal Knowledge Distillation-based Human Trajectory Forecasting

Jaewoo Jeong, Seohee Lee, Daehee Park, Giwon Lee, Kuk-Jin Yoon

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV

Comments Accepted to CVPR 2025

Progress-Aware Video Frame Captioning

Zihui Xue, Joungbin An, Xitong Yang, Kristen Grauman

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.CV

Comments Accepted by CVPR 2025, Project website: https://vision.cs.utexas.edu/projects/ProgressCaptioner/

ORION: A Holistic End-to-End Autonomous Driving Framework by Vision-Language Instructed Action Generation

Haoyu Fu, Diankun Zhang, Zongchuang Zhao, Jianfeng Cui, Dingkang Liang, Chong Zhang, Dingyuan Zhang, Hongwei Xie, Bing Wang, Xiang Bai

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

A Self-Correcting Vision-Language-Action Model for Fast and Slow System Manipulation

Chenxuan Li, Jiaming Liu, Guanqun Wang, Xiaoqi Li, Sixiang Chen, Liang Heng, Chuyan Xiong, Jiaxin Ge, Renrui Zhang, Kaichen Zhou, Shanghang Zhang

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV

Collaborative Instance Object Navigation: Leveraging Uncertainty-Awareness to Minimize Human-Agent Dialogues

Francesco Taioli, Edoardo Zorzi, Gianni Franchi, Alberto Castellini, Alessandro Farinelli, Marco Cristani, Yiming Wang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

Comments https://intelligolabs.github.io/CoIN/

Zero-Shot Action Recognition in Surveillance Videos

Joao Pereira, Vasco Lopes, David Semedo, Joao Neves

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling

Tsu-Jui Fu, Xin Eric Wang, Matthew Peterson, Scott Grafton, Miguel Eckstein, William Yang Wang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments ECCV'20 (Spotlight)

MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Models

Han Zhao, Wenxuan Song, Donglin Wang, Xinyang Tong, Pengxiang Ding, Xuelian Cheng, Zongyuan Ge

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.AI

Comments Accepted by ICRA 2025

VideoPhy-2: A Challenging Action-Centric Physical Commonsense Evaluation in Video Generation

Hritik Bansal, Clark Peng, Yonatan Bitton, Roman Goldenberg, Aditya Grover, Kai-Wei Chang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments 41 pages, 33 Figures

VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model

Jiani Zheng, Lu Wang, Fangkai Yang, Chaoyun Zhang, Lingrui Mei, Wenjie Yin, Qingwei Lin, Dongmei Zhang, Saravan Rajmohan, Qi Zhang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.LG

Comments 20pages,5 figures

Digi-Q: Learning Q-Value Functions for Training Device-Control Agents

Hao Bai, Yifei Zhou, Li Erran Li, Sergey Levine, Aviral Kumar

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.LG

Comments Accepted to ICLR 2025

PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC

Haowei Liu, Xi Zhang, Haiyang Xu, Yuyang Wanyan, Junyang Wang, Ming Yan, Ji Zhang, Chunfeng Yuan, Changsheng Xu, Weiming Hu, Fei Huang

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.CV

Comments 14 pages, 7 figures

NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistants

Yiran Qin, Ao Sun, Yuze Hong, Benyou Wang, Ruimao Zhang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Accepted to ICRA2025