视觉大模型 / VLM

VisioPath: Vision-Language Enhanced Model Predictive Control for Safe Autonomous Navigation in Mixed Traffic

Shanting Wang, Panagiotis Typaldos, Chenjun Li, Andreas A. Malikopoulos

机构 * System Engineering Program, Cornell University（Cornell大学系统工程项目） ； School of Civil and Environmental Engineering, Cornell University（Cornell大学土木与环境工程学院） ； School of Electrical and Computer Engineering, Cornell University（Cornell大学电气与计算机工程学院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)

DyNaVLM: Zero-Shot Vision-Language Navigation System with Dynamic Viewpoints and Self-Refining Graph Memory

Zihe Ji, Huangxuan Lin, Yue Gao

机构 * SJTU Paris Elite Institute of Technology, Shanghai Jiao Tong University（上海交通大学巴黎精英技术研究所） ； Department of Automation, Shanghai Jiao Tong University（上海交通大学自动化系） ； MoE Key Lab of Artificial Intelligence and AI Institute, Shanghai Jiao Tong University（上海交通大学人工智能与AI研究所） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)

Can Pretrained Vision-Language Embeddings Alone Guide Robot Navigation?

Nitesh Subedi, Adam Haroon, Shreyan Ganguly, Samuel T. K. Tetteh, Prajwal Koirala, Cody Fleming, Soumik Sarkar

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)

Comments 6 figures, 2 tables, Accepted to Robotics: Science and Systems (RSS) 2025 Workshop on Robot Planning in the Era of Foundation Models (FM4RoboPlan)

Atomic-to-Compositional Generalization for Mobile Agents with A New Benchmark and Scheduling System

Yuan Guo, Tingjia Miao, Zheng Wu, Pengzhou Cheng, Ming Zhou, Zhuosheng Zhang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；multimodal large language model(abstract)

Multimodal Spatial Language Maps for Robot Navigation and Manipulation

Chenguang Huang, Oier Mees, Andy Zeng, Wolfram Burgard

机构 * University of Technology Nuremberg（图恩堡技术大学） ； UC Berkeley（伯克利大学） ； Google Research（谷歌研究） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments accepted to International Journal of Robotics Research (IJRR). 24 pages, 18 figures. The paper contains texts from VLMaps(arXiv:2210.05714) and AVLMaps(arXiv:2303.07522). The project page is https://mslmaps.github.io/

BeliefMapNav: 3D Voxel-Based Belief Map for Zero-Shot Object Navigation

Zibo Zhou, Yue Hu, Lingkai Zhang, Zonglin Li, Siheng Chen

机构 * Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)

MiMo-VL Technical Report

Core Team, Zihao Yue, Zhenru Lin, Yifan Song, Weikun Wang, Shuhuai Ren, Shuhao Gu, Shicheng Li, Peidian Li, Liang Zhao, Lei Li, Kainan Bao, Hao Tian, Hailin Zhang, Gang Wang, Dawei Zhu, Cici, Chenhong He, Bowen Ye, Bowen Shen, Zihan Zhang, Zihan Jiang, Zhixian Zheng, Zhichao Song, Zhenbo Luo, Yue Yu, Yudong Wang, Yuanyuan Tian, Yu Tu, Yihan Yan, Yi Huang, Xu Wang, Xinzhe Xu, Xingchen Song, Xing Zhang, Xing Yong, Xin Zhang, Xiangwei Deng, Wenyu Yang, Wenhan Ma, Weiwei Lv, Weiji Zhuang, Wei Liu, Sirui Deng, Shuo Liu, Shimao Chen, Shihua Yu, Shaohui Liu, Shande Wang, Rui Ma, Qiantong Wang, Peng Wang, Nuo Chen, Menghang Zhu, Kangyang Zhou, Kang Zhou, Kai Fang, Jun Shi, Jinhao Dong, Jiebao Xiao, Jiaming Xu, Huaqiu Liu, Hongshen Xu, Heng Qu, Haochen Zhao, Hanglong Lv, Guoan Wang, Duo Zhang, Dong Zhang, Di Zhang, Chong Ma, Chang Liu, Can Cai, Bingquan Xia

机构 * LLM-Core ； Xiaomi（小米） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)

Comments 32 pages

LifeIR at the NTCIR-18 Lifelog-6 Task

Jiahan Chen, Da Li, Keping Bi

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agents

Yunseok Jang, Yeda Song, Sungryull Sohn, Lajanugen Logeswaran, Tiange Luo, Dong-Ki Kim, Kyunghoon Bae, Honglak Lee

机构 * University of Michigan（密歇根大学） ； LG AI Research（LG人工智能研究） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments CVPR 2025

BadNAVer: Exploring Jailbreak Attacks On Vision-and-Language Navigation

Wenqi Lyu, Zerui Li, Yanyuan Qiao, Qi Wu

机构 * Australian Institute for Machine Learning, University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

Comments 8 pages, 4 figures

VTLA: Vision-Tactile-Language-Action Model with Preference Learning for Insertion Manipulation

Chaofan Zhang, Peng Hao, Xiaoge Cao, Xiaoshuai Hao, Shaowei Cui, Shuo Wang

机构 * State Key Laboratory of Multimodal Artificial Intelligence Systems, Institute of Automation, Chinese Academy of Sciences（多模态人工智能系统国家重点实验室，自动化研究所，中国科学院） ； Samsung R&D Institute China–Beijing（三星中国北京研发中心） ； Beijing Academy of Artificial Intelligence（北京人工智能研究院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)

UAV-VLPA*: A Vision-Language-Path-Action System for Optimal Route Generation on a Large Scales

Oleg Sautenkov, Aibek Akhmetkazy, Yasheerah Yaqoot, Muhammad Ahsan Mustafa, Grik Tadevosyan, Artem Lykov, Dzmitry Tsetserukou

机构 * Intelligent Space Robotics Laboratory, Center for Digital Engineering, Skolkovo Institute of Science and Technology（智能空间机器人实验室，数字工程中心，斯克尔科沃科学与技术研究所） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract)；visual language model(abstract)

Comments arXiv admin note: text overlap with arXiv:2501.05014

Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success

Moo Jin Kim, Chelsea Finn, Percy Liang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted to Robotics: Science and Systems (RSS) 2025. Project website: https://openvla-oft.github.io/

SceneScout: Towards AI Agent-driven Access to Street View Imagery for Blind Users

Gaurav Jain, Leah Findlater, Cole Gleason

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents

Saaket Agashe, Kyle Wong, Vincent Tu, Jiachen Yang, Ang Li, Xin Eric Wang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 18 pages, 13 figures, 8 tables

DeskVision: Large Scale Desktop Region Captioning for Advanced GUI Agents

Yibin Xu, Liang Yang, Hao Chen, Hua Wang, Zhi Chen, Yaohua Tang

专题命中 GUI与屏幕智能体 ：visual language model(abstract)；grounding(abstract)

AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, Tao Yu

专题命中 GUI与屏幕智能体 ：VLM(abstract)；grounding(abstract)

Comments ICLR2025 Spotlight https://agenttrek.github.io

Magma: A Foundation Model for Multimodal AI Agents

Jianwei Yang, Reuben Tan, Qianhui Wu, Ruijie Zheng, Baolin Peng, Yongyuan Liang, Yu Gu, Mu Cai, Seonghyeon Ye, Joel Jang, Yuquan Deng, Lars Liden, Jianfeng Gao

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 29 pages, 16 figures, technical report from MSR

Leveraging Multimodal LLM for Inspirational User Interface Search

Seokhyeon Park, Yumin Song, Soohyun Lee, Jaeyoung Kim, Jinwook Seo

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

Comments In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (CHI '25)

Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agents

Yifei Zhou, Qianlan Yang, Kaixiang Lin, Min Bai, Xiong Zhou, Yu-Xiong Wang, Sergey Levine, Erran Li

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

WEPO: Web Element Preference Optimization for LLM-based Web Navigation

Jiarun Liu, Jia Hao, Chunhong Zhang, Zheng Hu

专题命中 GUI与屏幕智能体 ：visual language model(abstract)；grounding(abstract)

Comments Published at AAAI 2025

Seeing is Believing: Vision-driven Non-crash Functional Bug Detection for Mobile Apps

Zhe Liu, Cheng Li, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Yawen Wang, Jun Hu, Qing Wang

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)

CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation

Qixiu Li, Yaobo Liang, Zeyu Wang, Lin Luo, Xi Chen, Mozheng Liao, Fangyun Wei, Yu Deng, Sicheng Xu, Yizhong Zhang, Xiaofan Wang, Bei Liu, Jianlong Fu, Jianmin Bao, Dong Chen, Yuanchun Shi, Jiaolong Yang, Baining Guo

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Project Webpage: https://cogact.github.io/

OK-Robot: What Really Matters in Integrating Open-Knowledge Models for Robotics

Peiqi Liu, Yaswanth Orru, Jay Vakil, Chris Paxton, Nur Muhammad Mahi Shafiullah, Lerrel Pinto

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Github repo: https://github.com/ok-robot/ok-robot

VLN-Game: Vision-Language Equilibrium Search for Zero-Shot Semantic Navigation

Bangguo Yu, Yuzhen Liu, Lei Han, Hamidreza Kasaei, Tingguang Li, Ming Cao

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；vision language model(abstract)

Comments 15 pages, 9 figures

VoxAct-B: Voxel-Based Acting and Stabilizing Policy for Bimanual Manipulation

I-Chun Arthur Liu, Sicheng He, Daniel Seita, Gaurav Sukhatme

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted to the Conference on Robot Learning (CoRL) 2024

Navi2Gaze: Leveraging Foundation Models for Navigation and Target Gazing

Jun Zhu, Zihao Du, Haotian Xu, Fengbo Lan, Zilong Zheng, Bo Ma, Shengjie Wang, Tao Zhang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)

OmniParser for Pure Vision Based GUI Agent

Yadong Lu, Jianwei Yang, Yelong Shen, Ahmed Awadallah

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Malicious Path Manipulations via Exploitation of Representation Vulnerabilities of Vision-Language Navigation Systems

Chashi Mahiul Islam, Shaeke Salman, Montasir Shams, Xiuwen Liu, Piyush Kumar

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 8 pages, 5 figures. This paper has been accepted for publication at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) 2024

ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation

Kaiwen Zhou, Kaizhi Zheng, Connor Pryor, Yilin Shen, Hongxia Jin, Lise Getoor, Xin Eric Wang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV、cs.AI、cs.LG