视觉大模型 / VLM

Benchmarking the Generality of Vision-Language-Action Models

对视觉-语言-动作模型通用性的基准测试

Pranav Guruprasad, Sudipta Chowdhury, Harsh Sikka, Mridul Sharma, Helen Lu, Sean Rivera, Aryan Khurana, Hangliang Ren, Yangyue Wang

机构 * Manifold Research ； Metarch AI ； Georgia Tech（佐治亚理工学院） ； Tufts University（塔夫茨大学） ； Northeastern University（东北大学） ； Birla Institute of Technology and Science, Pilani（比拉理工学院，帕利尼） ； Institute for Research and Innovation in Intelligent Systems (IRIIS)（智能系统研究与创新研究所） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；grounding(abstract)；分类 cs.LG

AI总结 本文提出MultiNet v1.0基准，评估视觉-语言-动作模型在六个基础能力领域的跨领域泛化能力，发现现有模型在未见领域和模态转移时表现显著退化。

Comments 23 pages, 7 figures, and 1 table

Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy

视觉与语言动作政策的贝叶斯分解：看见以行动，提示以指定

Kechun Xu, Zhenjie Zhu, Anzhe Chen, Shuqi Zhao, Qing Huang, Yifei Yang, Haojian Lu, Rong Xiong, Masayoshi Tomizuka, Yue Wang

机构 * Zhejiang University（浙江大学） ； UC Berkeley（伯克利大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

AI总结 本文提出BayesVLA，通过贝叶斯分解策略，解决VLA模型中因模态不平衡导致的灾难性遗忘问题，提升泛化能力和指令遵循性能。

Lite VLA: Efficient Vision-Language-Action Control on CPU-Bound Edge Robots

Justin Williams, Kishor Datta Gupta, Roy George, Mrinmoy Sarkar

机构 * Department of Cyber-Physical Systems, Clark Atlanta University（克雷克阿特拉大学计算机物理系统系） ； Siemens Corporation（西门子公司） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation

Zichen Zhu, Hao Tang, Yansi Li, Dingye Liu, Hongshen Xu, Kunyao Lan, Danyang Zhang, Yixuan Jiang, Hao Zhou, Chenrun Wang, Situo Zhang, Liangtai Sun, Yixiao Wang, Yuheng Sun, Lu Chen, Kai Yu

机构 * Shanghai Jiao Tong University（上海交通大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI

Comments NAACL 2025 Demo Track [code] https://github.com/OpenDFM/MobA [dataset] https://huggingface.co/datasets/OpenDFM/MobA-MobBench

CronusVLA: Towards Efficient and Robust Manipulation via Multi-Frame Vision-Language-Action Modeling

Hao Li, Shuai Yang, Yilun Chen, Xinyi Chen, Xiaoda Yang, Yang Tian, Hanqing Wang, Tai Wang, Dahua Lin, Feng Zhao, Jiangmiao Pang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments 39 pages, 24 figures

Hierarchical Language Models for Semantic Navigation and Manipulation in an Aerial-Ground Robotic System

Haokun Liu, Zhaoqi Ma, Yunong Li, Junichiro Sugihara, Yicheng Chen, Jinjie Li, Moju Zhao

机构 * DRAGON Lab at Department of Mechanical Engineering, The University of Tokyo（东京大学机械工程系DRAGON实验室） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

Comments 18 pages, 10 figures

Journal ref Advanced Intelligent Systems, Oct. 2025

DaMo: Data Mixing Optimizer in Fine-tuning Multimodal LLMs for Mobile Phone Agents

Kai Shi, Jun Yang, Ni Yang, Binqiang Pan, Qingsong Xie, Chao Zhang, Zhenyu Yang, Tianhuang Su, Haonan Lu

机构 * OPPO AI Center（OPPO人工智能中心） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

StarBench: A Turn-Based RPG Benchmark for Agentic Multimodal Decision-Making and Information Seeking

Haoran Zhang, Chenhao Zhu, Sicong Guo, Hanzhe Guo, Haiming Li, Donglin Yu

机构 * University of Michigan（密歇根大学） ； Stanford University（斯坦福大学） ； University of Illinois Urbana–Champaign（伊利诺伊大学厄巴纳-香槟分校） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)；分类 cs.AI

VLA-0: Building State-of-the-Art VLAs with Zero Modification

Ankit Goyal, Hugo Hadfield, Xuning Yang, Valts Blukis, Fabio Ramos

机构 * NVIDIA 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

ContextVLA: Vision-Language-Action Model with Amortized Multi-Frame Context

Huiwon Jang, Sihyun Yu, Heeseung Kwon, Hojin Jeon, Younggyo Seo, Jinwoo Shin

机构 * KAIST（韩国科学技术院） ； RLWRLD ； UC Berkeley（伯克利大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

Comments Project page: https://huiwon-jang.github.io/contextvla

Cross-Modal Content Optimization for Steering Web Agent Preferences

Tanqiu Jiang, Min Bai, Nikolaos Pappas, Yanjun Qi, Sandesh Swamy

机构 * Stony Brook University（石溪大学） ； AWS AI Labs（亚马逊人工智能实验室） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

Scaling Synthetic Task Generation for Agents via Exploration

Ram Ramrakhya, Andrew Szot, Omar Attia, Yuhao Yang, Anh Nguyen, Bogdan Mazoure, Zhe Gan, Harsh Agrawal, Alexander Toshev

机构 * Apple（苹果公司） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI

ReCogDrive: A Reinforced Cognitive Framework for End-to-End Autonomous Driving

Yongkang Li, Kaixin Xiong, Xiangyu Guo, Fang Li, Sixu Yan, Gangwei Xu, Lijun Zhou, Long Chen, Haiyang Sun, Bing Wang, Kun Ma, Guang Chen, Hangjun Ye, Wenyu Liu, Xinggang Wang

机构 * Huazhong University of Science and Technology（华中科技大学） ； Xiaomi EV（小米电动车） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning

Zimu Lu, Houxing Ren, Yunqiao Yang, Ke Wang, Zhuofan Zong, Junting Pan, Mingjie Zhan, Hongsheng Li

机构 * Multimedia Laboratory (MMLab)（多媒体实验室） 纠错

专题命中 GUI与屏幕智能体 ：VLM(abstract)；visual language model(abstract)；分类 cs.AI

DORAEMON: Decentralized Ontology-aware Reliable Agent with Enhanced Memory Oriented Navigation

Tianjun Gu, Linfeng Li, Xuhong Wang, Chenghua Gong, Jingyu Gong, Zhizhong Zhang, Yuan Xie, Lizhuang Ma, Xin Tan

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

XBOUND: Exploring Capability Boundaries of Device-Control Agents at the State Level

Shaoqing Zhang, Kehai Chen, Zhuosheng Zhang, Rumei Li, Rongxiang Weng, Yang Xiang, Min Zhang

机构 * Harbin Institute of Technology（哈尔滨工业大学） ； Pengcheng Laboratory（鹏城实验室） ； Shanghai Jiao Tong University（上海交通大学） ； Meituan（美团） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)；分类 cs.AI

UIPro: Unleashing Superior Interaction Capability For GUI Agents

Hongxin Li, Jingran Su, Jingfan Chen, Zheng Ju, Yuntao Chen, Qing Li, Zhaoxiang Zhang

机构 * University of Chinese Academy of Sciences (UCAS)（中国科学院大学） ； New Laboratory of Pattern Recognition (NLPR), CASIA（中国科学院自动化所模式识别新实验室） ； State Key Laboratory of Multimodal Artificial Intelligence Systems (MAIS), CASIA（中国科学院多模态人工智能系统国家重点实验室） ； Hong Kong Institute of Science & Innovation, CASIA（中国科学院香港创新科学研究院） ； PolyU ； Shanghai Artificial Intelligence Laboratory（上海人工智能实验室） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；grounding(abstract)；分类 cs.CV

Comments Accepted to ICCV 2025

AppAgent v2: Advanced Agent for Flexible Mobile Interactions

Yanda Li, Chi Zhang, Wenjia Jiang, Wanqi Yang, Bin Fu, Pei Cheng, Xin Chen, Ling Chen, Yunchao Wei

机构 * University of Technology Sydney（悉尼技术大学） ； Tencent（腾讯） ； Beijing Jiaotong University（北京交通大学） ； Westlake University（西湖大学） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.AI

Seeing the Undefined: Chain-of-Action for Generative Semantic Labels

Meng Wei, Zhongnian Li, Peng Ying, Xinzheng Xu

机构 * 1 School of Computer Science ； Technology / School of Artificial Intelligence, China University of Mining ； Technology Xuzhou China 2 Mine Digitization Engineering Research Center of the Ministry of Education Xuzhou China ； Technology Xuzhou China 2 The State Key Laboratory of CAD\&CG, Zhejiang University Hangzhou China 3 Mine Digitization Engineering Research Center of the Ministry of Education Xuzhou China ； 2 Mine Digitization Engineering Research Center of the Ministry of Education ； 2 The State Key Laboratory of CAD\&CG, Zhejiang University ； 3 Mine Digitization Engineering Research Center of the Ministry of Education 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments 15 pages, 8 figures

Galaxea Open-World Dataset and G0 Dual-System VLA Model

Tao Jiang, Tianyuan Yuan, Yicheng Liu, Chenhao Lu, Jianning Cui, Xiao Liu, Shuiqi Cheng, Jiyang Gao, Huazhe Xu, Hang Zhao

机构 * Galaxea Team（Galaxea 团队） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments https://opengalaxea.github.io/G0/

SafeBimanual: Diffusion-based Trajectory Optimization for Safe Bimanual Manipulation

Haoyuan Deng, Wenkai Guo, Qianzhun Wang, Zhenyu Wu, Ziwei Wang

机构 * Nanyang Technological University（南洋理工大学） ； Beijing University of Posts and Telecommunications（北京邮电大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

Comments Project website is at: https://denghaoyuan123.github.io/SafeBimanip/

AeroDuo: Aerial Duo for UAV-based Vision and Language Navigation

Ruipu Wu, Yige Zhang, Jinyu Chen, Linjiang Huang, Shifeng Zhang, Xu Zhou, Liang Wang, Si Liu

机构 * Beihang University（北京航空航天大学） ； Sangfor Technologies Inc.（深信服科技有限公司） ； Institute of Automation, Chinese Academy of Sciences（中国科学院自动化研究所） 纠错

专题命中 GUI与屏幕智能体 ：grounding(abstract)；multimodal large language model(abstract)；分类 cs.CV

Comments Accepted by ACM MM 2025

UI-Venus Technical Report: Building High-performance UI Agents with RFT

Zhangxuan Gu, Zhengwen Zeng, Zhenyu Xu, Xingran Zhou, Shuheng Shen, Yunfei Liu, Beitong Zhou, Changhua Meng, Tianyu Xia, Weizhi Chen, Yue Wen, Jingya Dou, Fei Tang, Jinzhen Lin, Yulin Liu, Zhenlin Guo, Yichen Gong, Heng Jia, Changlong Gao, Yuan Guo, Yong Deng, Zhenyu Guo, Liang Chen, Weiqiang Wang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；multimodal large language model(abstract)；分类 cs.CV

Rhetorical Text-to-Image Generation via Two-layer Diffusion Policy Optimization

Yuxi Zhang, Yueting Li, Xinyu Du, Sibo Wang

机构 * The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）） ； University of California, Berkeley（加州大学伯克利分校） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Cross-Modal Dual-Causal Learning for Long-Term Action Recognition

Xu Shaowu, Jia Xibin, Gao Junyu, Sun Qianmei, Chang Jing, Fan Chao

机构 * Beijing University of Technology（北京理工大学） ； Chinese Academy of Sciences（中国科学院） ； Capital Medical University（首都医科大学） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

ERMV: Editing 4D Robotic Multi-view images to enhance embodied agents

Chang Nie, Guangming Wang, Zhe Lie, Hesheng Wang

机构 * School of Automation and Intelligent Sensing, Shanghai Jiao Tong University and Key Laboratory of System Control and Information Processing, Ministry of Education of China（自动化与智能感知学院，上海交通大学；系统控制与信息处理重点实验室，中华人民共和国教育部） 纠错

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

Tactile-VLA: Unlocking Vision-Language-Action Model's Physical Knowledge for Tactile Generalization

Jialei Huang, Shuo Wang, Fanqi Lin, Yihang Hu, Chuan Wen, Yang Gao

专题命中 GUI与屏幕智能体 ：VLM(abstract)；grounding(abstract)；分类 cs.LG

Adaptive Framework for Ambient Intelligence in Rehabilitation Assistance

Gábor Baranyi, Zsolt Csibi, Kristian Fenech, Áron Fóthi, Zsófia Gaál, Joul Skaf, András Lőrincz

机构 * Emineo Private Hospital（埃米尼欧私人医院） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；VLM(abstract)；分类 cs.AI

Comments The paper has been submitted to a journal and waiting for review

RoboMonkey: Scaling Test-Time Sampling and Verification for Vision-Language-Action Models

Jacky Kwok, Christopher Agia, Rohan Sinha, Matt Foutter, Shulu Li, Ion Stoica, Azalia Mirhoseini, Marco Pavone

机构 * Stanford University（斯坦福大学） ； UC Berkeley（加州大学伯克利分校） ； NVIDIA Research（NVIDIA研究） 纠错

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；VLM(abstract)；分类 cs.AI

Balancing Performance and Efficiency in Zero-shot Robotic Navigation

Dmytro Kuzmenko, Nadiya Shvai

机构 * Department of Multimedia Systems, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine（多媒体系统系，基辅-莫希拉学院国家大学，乌克兰基辅） ； Department of Mathematics, National University of Kyiv-Mohyla Academy, Kyiv, Ukraine（数学系，基辅-莫希拉学院国家大学，乌克兰基辅） 纠错

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Submitted to ICTERI 2024 Posters Track

Journal ref ICTERI 2024: Communications in Computer and Information Science, vol. 2020, pp. 370-381, Springer, 2025