视觉大模型 / VLM

Advancing Vision-based Human Action Recognition: Exploring Vision-Language CLIP Model for Generalisation in Domain-Independent Tasks

Utkarsh Shandilya, Marsha Mariya Kappan, Sanyam Jain, Vijeta Sharma

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Robotic Visual Instruction

Yanbang Li, Ziyang Gong, Haoyang Li, Xiaoqi Huang, Haolan Kang, Guangping Bai, Xianzheng Ma

机构 * Imperial College London（伦敦帝国理工学院） ； Shanghai AI Laboratory（上海人工智能实验室） ； UC San Diego（加州大学圣地亚哥分校） ； VIVO ； South China University of Technology（华南理工大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Project website: https://robotic-visual-instruction.github.io/

Hi Robot: Open-Ended Instruction Following with Hierarchical Vision-Language-Action Models

Lucy Xiaoyang Shi, Brian Ichter, Michael Equi, Liyiming Ke, Karl Pertsch, Quan Vuong, James Tanner, Anna Walling, Haohuan Wang, Niccolo Fusai, Adrian Li-Bell, Danny Driess, Lachy Groom, Sergey Levine, Chelsea Finn

机构 * Stanford University（斯坦福大学） ； University of California, Berkeley（加州大学伯克利分校） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI、cs.LG

Comments ICML 2025

OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis

Qiushi Sun, Kanzhi Cheng, Zichen Ding, Chuanyang Jin, Yian Wang, Fangzhi Xu, Zhenyu Wu, Chengyou Jia, Liheng Chen, Zhoumianze Liu, Ben Kao, Guohao Li, Junxian He, Yu Qiao, Zhiyong Wu

机构 * Shanghai AI Laboratory（上海人工智能实验室） ； The University of Hong Kong（香港大学） ； Johns Hopkins University（约翰霍普金斯大学） ； Shanghai Jiao Tong University（上海交通大学） ； University of Oxford（牛津大学） ； Hong Kong University of Science and Technology（香港科技大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments ACL 2025 Camera Ready

Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation

Xinzge Gao, Chuanrui Hu, Bin Chen, Teng Li

机构 * Anhui University（安徽大学） ； Qihoo360（奇虎360） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning

Zhong Zhang, Yaxi Lu, Yikun Fu, Yupeng Huo, Shenzhi Yang, Yesai Wu, Han Si, Xin Cong, Haotian Chen, Yankai Lin, Jie Xie, Wei Zhou, Wang Xu, Yuanheng Zhang, Zhou Su, Zhongwu Zhai, Xiaoming Liu, Yudong Mei, Jianming Xu, Hongyan Tian, Chongyi Wang, Chi Chen, Yuan Yao, Zhiyuan Liu, Maosong Sun

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI

Comments Updated results in Table 2 and Table 3; The project is available at https://github.com/OpenBMB/AgentCPM-GUI

Benchmarking Vision, Language, & Action Models in Procedurally Generated, Open Ended Action Environments

Pranav Guruprasad, Yangyue Wang, Sudipta Chowdhury, Harshvardhan Sikka, Paul Pu Liang

机构 * Manifold Research ； MIT（麻省理工学院） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV、cs.LG

Comments 16 pages, 26 figures

DPO Learning with LLMs-Judge Signal for Computer Use Agents

Man Luo, David Cobbley, Xin Su, Shachar Rosenman, Vasudev Lal, Shao-Yen Tseng, Phillip Howard

机构 * Intel（英特尔） ； Thoughtworks 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction

Shravan Nayak, Xiangru Jian, Kevin Qinghong Lin, Juan A. Rodriguez, Montek Kalsi, Rabiul Awal, Nicolas Chapados, M. Tamer Özsu, Aishwarya Agrawal, David Vazquez, Christopher Pal, Perouz Taslakian, Spandana Gella, Sai Rajeswar

机构 * Mila - Quebec AI Institute（魁北克AI研究所） ； University of Waterloo（滑铁卢大学） ； National University of Singapore（新加坡国立大学） ； CIFAR AI Chair（CIFAR人工智能职位） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI

Comments This paper has been accepted to the 41st International Conference on Machine Learning (ICML 2025)

Phoenix: A Motion-based Self-Reflection Framework for Fine-grained Robotic Action Correction

Wenke Xia, Ruoxuan Feng, Dong Wang, Di Hu

机构 * Gaoling School of Artificial Intelligence, Renmin University of China（中国人民大学北京耿丽人工智能学院） ； Shanghai AI Laboratory（上海人工智能实验室） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by CVPR2025

Breaking the Data Barrier -- Building GUI Agents Through Task Generalization

Junlei Zhang, Zichen Ding, Chang Ma, Zijie Chen, Qiushi Sun, Zhenzhong Lan, Junxian He

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.CV、cs.AI

Comments 24 pages, 11 figures

A Taxonomy of Self-Handover

Naoki Wake, Atsushi Kanehira, Kazuhiro Sasabuchi, Jun Takamatsu, Katsushi Ikeuchi

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments 8 pages, 8 figures, 1 table, Last updated on April 7th, 2025

SmolVLM: Redefining small and efficient multimodal models

Andrés Marafioti, Orr Zohar, Miquel Farré, Merve Noyan, Elie Bakouch, Pedro Cuenca, Cyril Zakka, Loubna Ben Allal, Anton Lozhkov, Nouamane Tazi, Vaibhav Srivastav, Joshua Lochner, Hugo Larcher, Mathieu Morlon, Lewis Tunstall, Leandro von Werra, Thomas Wolf

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

STEVE: A Step Verification Pipeline for Computer-use Agent Training

Fanbin Lu, Zhisheng Zhong, Ziqin Wei, Shu Liu, Chi-Wing Fu, Jiaya Jia

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

MP-GUI: Modality Perception with MLLMs for GUI Understanding

Ziwei Wang, Weizhi Chen, Leyang Yang, Sheng Zhou, Shengchu Zhao, Hanbei Zhan, Jiongchao Jin, Liangcheng Li, Zirui Shao, Jiajun Bu

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.CV、cs.AI

Comments Paper accepted to CVPR 2025

MoMa-Kitchen: A 100K+ Benchmark for Affordance-Grounded Last-Mile Navigation in Mobile Manipulation

Pingrui Zhang, Xianqiang Gao, Yuhan Wu, Kehui Liu, Dong Wang, Zhigang Wang, Bin Zhao, Yan Ding, Xuelong Li

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI

EdgeMoE: Empowering Sparse Large Language Models on Mobile Devices

Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, Mengwei Xu

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.AI、cs.LG

DistRL: An Asynchronous Distributed Reinforcement Learning Framework for On-Device Control Agents

Taiyi Wang, Zhihao Wu, Jianheng Liu, Jianye Hao, Jun Wang, Kun Shao

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments Paper and Appendix, 26 pages

Foundation Models for CPS-IoT: Opportunities and Challenges

Ozan Baris, Yizhuo Chen, Gaofeng Dong, Liying Han, Tomoyoshi Kimura, Pengrui Quan, Ruijie Wang, Tianchen Wang, Tarek Abdelzaher, Mario Bergés, Paul Pu Liang, Mani Srivastava

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Improving Vision-Language-Action Model with Online Reinforcement Learning

Yanjiang Guo, Jianke Zhang, Xiaoyu Chen, Xiang Ji, Yen-Jen Wang, Yucheng Hu, Jianyu Chen

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted to ICRA 2025

UI-TARS: Pioneering Automated GUI Interaction with Native Agents

Yujia Qin, Yining Ye, Junjie Fang, Haoming Wang, Shihao Liang, Shizuo Tian, Junda Zhang, Jiahao Li, Yunxin Li, Shijue Huang, Wanjun Zhong, Kuanye Li, Jiale Yang, Yu Miao, Woyu Lin, Longxiang Liu, Xu Jiang, Qianli Ma, Jingyu Li, Xiaojun Xiao, Kai Cai, Chuang Li, Yaowei Zheng, Chaolin Jin, Chen Li, Xiao Zhou, Minchao Wang, Haoli Chen, Zhaojian Li, Haihua Yang, Haifeng Liu, Feng Lin, Tao Peng, Xin Liu, Guang Shi

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI

YETI (YET to Intervene) Proactive Interventions by Multimodal AI Agents in Augmented Reality Tasks

Saptarashmi Bandyopadhyay, Vikas Bahirwani, Lavisha Aggarwal, Bhanu Guda, Lin Li, Andrea Colaco

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Preprint

Efficient Policy Adaptation with Contrastive Prompt Ensemble for Embodied Agents

Wonje Choi, Woo Kyung Kim, SeungHyun Kim, Honguk Woo

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted at NeurIPS 2023

Benchmarking Vision, Language, & Action Models on Robotic Learning Tasks

Pranav Guruprasad, Harshvardhan Sikka, Jaewoo Song, Yangyue Wang, Paul Pu Liang

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV、cs.LG

Comments 16 Pages, 10 Figures

ShowUI: One Vision-Language-Action Model for GUI Visual Agent

Kevin Qinghong Lin, Linjie Li, Difei Gao, Zhengyuan Yang, Shiwei Wu, Zechen Bai, Weixian Lei, Lijuan Wang, Mike Zheng Shou

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI

Comments Technical Report. Github: https://github.com/showlab/ShowUI

AdaptAgent: Adapting Multimodal Web Agents with Few-Shot Learning from Human Demonstrations

Gaurav Verma, Rachneet Kaur, Nishan Srishankar, Zhen Zeng, Tucker Balch, Manuela Veloso

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments 18 pages, 3 figures, an abridged version to appear in NeurIPS 2024 AFM Workshop

Dialog2Flow: Pre-training Soft-Contrastive Action-Driven Sentence Embeddings for Automatic Dialog Flow Extraction

Sergio Burdisso, Srikanth Madikeri, Petr Motlicek

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI、cs.LG

Comments Accepted to EMNLP 2024 main conference

Journal ref https://aclanthology.org/2024.emnlp-main.310/

AutoGLM: Autonomous Foundation Agents for GUIs

Xiao Liu, Bo Qin, Dongzhu Liang, Guang Dong, Hanyu Lai, Hanchen Zhang, Hanlin Zhao, Iat Long Iong, Jiadai Sun, Jiaqi Wang, Junjie Gao, Junjun Shan, Kangning Liu, Shudan Zhang, Shuntian Yao, Siyi Cheng, Wentao Yao, Wenyi Zhao, Xinghan Liu, Xinyi Liu, Xinying Chen, Xinyue Yang, Yang Yang, Yifan Xu, Yu Yang, Yujia Wang, Yulin Xu, Zehan Qi, Yuxiao Dong, Jie Tang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI、cs.LG

Keypoint Abstraction using Large Models for Object-Relative Imitation Learning

Xiaolin Fang, Bo-Ruei Huang, Jiayuan Mao, Jasmine Shone, Joshua B. Tenenbaum, Tomás Lozano-Pérez, Leslie Pack Kaelbling

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments CoRL LangRob Workshop, 2024

UI-JEPA: Towards Active Perception of User Intent through Onscreen User Activity

Yicheng Fu, Raviteja Anantha, Prabal Vashisht, Jianpeng Cheng, Etai Littwin

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI、cs.LG