视觉大模型 / VLM

SmartAgent: Chain-of-User-Thought for Embodied Personalized Agent in Cyber World

Jiaqi Zhang, Chen Gao, Liyuan Zhang, Yong Li, Hongzhi Yin

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

GUI Agents with Foundation Models: A Comprehensive Survey

Shuai Wang, Weiwen Liu, Jingxuan Chen, Yuqi Zhou, Weinan Gan, Xingshan Zeng, Yuhan Che, Shuai Yu, Xinlong Hao, Kun Shao, Bin Wang, Chuhan Wu, Yasheng Wang, Ruiming Tang, Jianye Hao

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI

HumanOmni: A Large Vision-Speech Language Model for Human-Centric Video Understanding

Jiaxing Zhao, Qize Yang, Yixing Peng, Detao Bai, Shimin Yao, Boyuan Sun, Xiang Chen, Shenghao Fu, Weixuan chen, Xihan Wei, Liefeng Bo

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV

Large Action Models: From Inception to Implementation

Lu Wang, Fangkai Yang, Chaoyun Zhang, Junting Lu, Jiaxu Qian, Shilin He, Pu Zhao, Bo Qiao, Ray Huang, Si Qin, Qisheng Su, Jiayi Ye, Yudi Zhang, Jian-Guang Lou, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Comments 25pages,12 figures

Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation

Senwei Xie, Hongyu Wang, Zhanqi Xiao, Ruiping Wang, Xilin Chen

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Large Language Models for Video Surveillance Applications

Ulindu De Silva, Leon Fernando, Billy Lau Pik Lik, Zann Koh, Sam Conrad Joyce, Belinda Yuen, Chau Yuen

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.CV

Comments Accepted for TENCON 2024

GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent

Kangjia Zhao, Jiahui Song, Leigang Sha, Haozhan Shen, Zhi Chen, Tiancheng Zhao, Xiubo Liang, Jianwei Yin

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI

MRWeb: An Exploration of Generating Multi-Page Resource-Aware Web Code from UI Designs

Yuxuan Wan, Yi Dong, Jingyu Xiao, Yintong Huo, Wenxuan Wang, Michael R. Lyu

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.AI

MobileFlow: A Multimodal LLM For Mobile GUI Agent

Songqin Nong, Jiali Zhu, Rui Wu, Jiongchao Jin, Shuo Shan, Xiutian Huang, Wenhao Xu

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV

Generative AI in Multimodal User Interfaces: Trends, Challenges, and Cross-Platform Adaptability

J. Bieniek, M. Rahouti, D. C. Verma

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI

Comments 13 pages, 4 figures

NavAgent: Multi-scale Urban Street View Fusion For UAV Embodied Vision-and-Language Navigation

Youzhi Liu, Fanglong Yao, Yuanchang Yue, Guangluan Xu, Xian Sun, Kun Fu

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Cognitive Planning for Object Goal Navigation using Generative AI Models

Arjun P S, Andrew Melnik, Gora Chand Nandi

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

EDGE: Enhanced Grounded GUI Understanding with Enriched Multi-Granularity Synthetic Data

Xuetian Chen, Hangcheng Li, Jiaqing Liang, Sihang Jiang, Deqing Yang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

Reliable Semantic Understanding for Real World Zero-shot Object Goal Navigation

Halil Utku Unlu, Shuaihang Yuan, Congcong Wen, Hao Huang, Anthony Tzes, Yi Fang

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.AI

Comments 16 pages, 7 figures, 2 tables

Perceive, Reflect, and Plan: Designing LLM Agent for Goal-Directed City Navigation without Instructions

Qingbin Zeng, Qinglong Yang, Shunan Dong, Heming Du, Liang Zheng, Fengli Xu, Yong Li

专题命中 GUI与屏幕智能体 ：LLaVA(abstract)；分类 cs.AI

Robots Can Multitask Too: Integrating a Memory Architecture and LLMs for Enhanced Cross-Task Robot Action Generation

Hassan Ali, Philipp Allgeuer, Carlo Mazzola, Giulia Belgiovine, Burak Can Kaplan, Lukáš Gajdošech, Stefan Wermter

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Towards Realistic UAV Vision-Language Navigation: Platform, Benchmark, and Methodology

Xiangyu Wang, Donglin Yang, Ziqin Wang, Hohin Kwan, Jinyu Chen, Wenjun Wu, Hongsheng Li, Yue Liao, Si Liu

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.CV

VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values

Zhe Hu, Yixiao Ren, Jing Li, Yu Yin

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.CV

Comments EMNLP 2024 Main Conference

LADEV: A Language-Driven Testing and Evaluation Platform for Vision-Language-Action Models in Robotic Manipulation

Zhijie Wang, Zhehua Zhou, Jiayang Song, Yuheng Huang, Zhan Shu, Lei Ma

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.AI

Comments 8 pages, 4 figures

Improving Zero-Shot ObjectNav with Generative Communication

Vishnu Sashank Dorbala, Vishnu Dutt Sharma, Pratap Tokekar, Dinesh Manocha

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

CLSP: High-Fidelity Contrastive Language-State Pre-training for Agent State Representation

Fuxian Huang, Qi Zhang, Shaopeng Zhai, Jie Wang, Tianyi Zhang, Haoran Zhang, Ming Zhou, Yu Liu, Yu Qiao

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.AI

Can VLMs Play Action Role-Playing Games? Take Black Myth Wukong as a Study Case

Peng Chen, Pi Bu, Jun Song, Yuan Gao, Bo Zheng

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.AI

Towards Unified Facial Action Unit Recognition Framework by Large Language Models

Guohong Hu, Xing Lan, Hanyu Jiang, Jiayi Lyu, Jian Xue

专题命中 GUI与屏幕智能体 ：LLaVA(abstract)；分类 cs.CV

RoboUniView: Visual-Language Model with Unified View Representation for Robotic Manipulation

Fanfan Liu, Feng Yan, Liming Zheng, Chengjian Feng, Yiyang Huang, Lin Ma

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Text-Enhanced Zero-Shot Action Recognition: A training-free approach

Massimo Bosetti, Shibingfeng Zhang, Benedetta Liberatori, Giacomo Zara, Elisa Ricci, Paolo Rota

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments accepted to ICPR 2024

Robot Utility Models: General Policies for Zero-Shot Deployment in New Environments

Haritheja Etukuru, Norihito Naka, Zijin Hu, Seungjae Lee, Julian Mehu, Aaron Edsinger, Chris Paxton, Soumith Chintala, Lerrel Pinto, Nur Muhammad Mahi Shafiullah

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.LG

Comments Project website https://robotutilitymodels.com

OpenVLA: An Open-Source Vision-Language-Action Model

Moo Jin Kim, Karl Pertsch, Siddharth Karamcheti, Ted Xiao, Ashwin Balakrishna, Suraj Nair, Rafael Rafailov, Ethan Foster, Grace Lam, Pannag Sanketi, Quan Vuong, Thomas Kollar, Benjamin Burchfiel, Russ Tedrake, Dorsa Sadigh, Sergey Levine, Percy Liang, Chelsea Finn

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.LG

Comments Website: https://openvla.github.io/

Multi-Modality Co-Learning for Efficient Skeleton-based Action Recognition

Jinfu Liu, Chen Chen, Mengyuan Liu

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV

Loc4Plan: Locating Before Planning for Outdoor Vision and Language Navigation

Huilin Tian, Jingke Meng, Wei-Shi Zheng, Yuan-Ming Li, Junkai Yan, Yunong Zhang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments arXiv admin note: text overlap with arXiv:2203.13838 by other authors

PARSE-Ego4D: Personal Action Recommendation Suggestions for Egocentric Videos

Steven Abreu, Tiffany D. Do, Karan Ahuja, Eric J. Gonzalez, Lee Payne, Daniel McDuff, Mar Gonzalez-Franco

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV