视觉大模型 / VLM

PALM: Predicting Actions through Language Models

Sanghwan Kim, Daoji Huang, Yongqin Xian, Otmar Hilliges, Luc Van Gool, Xi Wang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Prioritized Semantic Learning for Zero-shot Instance Navigation

Xinyu Sun, Lizhao Liu, Hongyan Zhi, Ronghe Qiu, Junwei Liang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Accepted by ECCV 2024

Local Action-Guided Motion Diffusion Model for Text-to-Motion Generation

Peng Jin, Hao Li, Zesen Cheng, Kehan Li, Runyi Yu, Chang Liu, Xiangyang Ji, Li Yuan, Jie Chen

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments Accepted by ECCV 2024

MobileExperts: A Dynamic Tool-Enabled Agent Team in Mobile Devices

Jiayi Zhang, Chuang Zhao, Yihan Zhao, Zhaoyang Yu, Ming He, Jianping Fan

专题命中 GUI与屏幕智能体 ：visual language model(abstract)；分类 cs.AI

Tell Me Where You Are: Multimodal LLMs Meet Place Recognition

Zonglin Lyu, Juexiao Zhang, Mingxuan Lu, Yiming Li, Chen Feng

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.CV

Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction

Danyang Zhang, Zhennan Shen, Rui Xie, Situo Zhang, Tianbao Xie, Zihan Zhao, Siyuan Chen, Lu Chen, Hongshen Xu, Ruisheng Cao, Kai Yu

专题命中 GUI与屏幕智能体 ：vision language model(abstract)；分类 cs.AI

Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration

Junyang Wang, Haiyang Xu, Haitao Jia, Xi Zhang, Ming Yan, Weizhou Shen, Ji Zhang, Fei Huang, Jitao Sang

专题命中 GUI与屏幕智能体 ：MLLM(abstract)；分类 cs.CV

Comments 22 pages, 11 figures, 10 Tables

UFO: A UI-Focused Agent for Windows OS Interaction

Chaoyun Zhang, Liqun Li, Shilin He, Xu Zhang, Bo Qiao, Si Qin, Minghua Ma, Yu Kang, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, Qi Zhang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

LOC-ZSON: Language-driven Object-Centric Zero-Shot Object Retrieval and Navigation

Tianrui Guan, Yurou Yang, Harry Cheng, Muyuan Lin, Richard Kim, Rajasimman Madhivanan, Arnie Sen, Dinesh Manocha

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV

Comments Accepted to ICRA 2024

Unifying Global and Local Scene Entities Modelling for Precise Action Spotting

Kim Hoang Tran, Phuc Vuong Do, Ngoc Quoc Ly, Ngan Le

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Accepted to IJCNN 2024

Can an Embodied Agent Find Your "Cat-shaped Mug"? LLM-Guided Exploration for Zero-Shot Object Navigation

Vishnu Sashank Dorbala, James F. Mullen, Dinesh Manocha

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Comments 10 pages

Journal ref IEEE Robotics and Automation Letters 9.5 (2024) 4083-4090

Towards Objectively Benchmarking Social Intelligence for Language Agents at Action Level

Chenxu Wang, Bin Dai, Huaping Liu, Baoyuan Wang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

OST: Refining Text Knowledge with Optimal Spatio-Temporal Descriptor for General Video Recognition

Tongjia Chen, Hongshan Yu, Zhengeng Yang, Zechuan Li, Wei Sun, Chen Chen

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Technical report. Project Page: https://tomchen-ctj.github.io/OST/

MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perception

Yiran Qin, Enshen Zhou, Qichang Liu, Zhenfei Yin, Lu Sheng, Ruimao Zhang, Yu Qiao, Jing Shao

专题命中 GUI与屏幕智能体 ：multimodal large language model(abstract)；分类 cs.CV

Comments Accepted to CVPR2024

Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation

Razvan-George Pasca, Alexey Gavryushin, Muhammad Hamza, Yen-Ling Kuo, Kaichun Mo, Luc Van Gool, Otmar Hilliges, Xi Wang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

One-shot Imitation in a Non-Stationary Environment via Multi-Modal Skill

Sangwoo Shin, Daehee Lee, Minjong Yoo, Woo Kyung Kim, Honguk Woo

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

Comments ICML-2023 Camera Ready Version

Target-Grounded Graph-Aware Transformer for Aerial Vision-and-Dialog Navigation

Yifei Su, Dong An, Yuan Xu, Kehan Chen, Yan Huang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments 1st Place Solution for the AVDN Challenge in ICCV CLVL 2023

VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation

Naoki Yokoyama, Sehoon Ha, Dhruv Batra, Jiuguang Wang, Bernadette Bucher

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.AI

TENT: Connect Language Models with IoT Sensors for Zero-Shot Activity Recognition

Yunjiao Zhou, Jianfei Yang, Han Zou, Lihua Xie

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Preprint manuscript in submission

PromptInfuser: How Tightly Coupling AI and UI Design Impacts Designers' Workflows

Savvas Petridis, Michael Terry, Carrie J. Cai

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Videoprompter: an ensemble of foundational models for zero-shot video understanding

Adeel Yousaf, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan, Mubarak Shah

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Video Action Recognition with Attentive Semantic Units

Yifei Chen, Dapeng Chen, Ruijin Liu, Hao Li, Wei Peng

专题命中 GUI与屏幕智能体 ：VLM(abstract)；分类 cs.CV

Comments Accepted at ICCV 2023

Journal ref Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 10170-10180

RIC: Rotate-Inpaint-Complete for Generalizable Scene Reconstruction

Isaac Kasahara, Shubham Agrawal, Selim Engin, Nikhil Chavan-Dafle, Shuran Song, Volkan Isler

专题命中 GUI与屏幕智能体 ：visual language model(abstract)；分类 cs.CV

Masked Diffusion with Task-awareness for Procedure Planning in Instructional Videos

Fen Fang, Yun Liu, Ali Koksal, Qianli Xu, Joo-Hwee Lim

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments 7 pages (main text excluding references), 3 figures, 7 tables

DiverseMotion: Towards Diverse Human Motion Generation via Discrete Diffusion

Yunhong Lou, Linchao Zhu, Yaxiong Wang, Xiaohan Wang, Yi Yang

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments 12 pages, 7 figures

Kefa: A Knowledge Enhanced and Fine-grained Aligned Speaker for Navigation Instruction Generation

Haitian Zeng, Xiaohan Wang, Wenguan Wang, Yi Yang

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.CV

Comments 10 pages, 4 figures

Referring to Screen Texts with Voice Assistants

Shruti Bhargava, Anand Dhoot, Ing-Marie Jonsson, Hoang Long Nguyen, Alkesh Patel, Hong Yu, Vincent Renkens

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Comments 7 pages, Accepted to ACL Industry Track 2023

InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language

Zhaoyang Liu, Yinan He, Wenhai Wang, Weiyun Wang, Yi Wang, Shoufa Chen, Qinglong Zhang, Zeqiang Lai, Yang Yang, Qingyun Li, Jiashuo Yu, Kunchang Li, Zhe Chen, Xue Yang, Xizhou Zhu, Yali Wang, Limin Wang, Ping Luo, Jifeng Dai, Yu Qiao

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments Technical Report

Filtering, Distillation, and Hard Negatives for Vision-Language Pre-Training

Filip Radenovic, Abhimanyu Dubey, Abhishek Kadian, Todor Mihaylov, Simon Vandenhende, Yash Patel, Yi Wen, Vignesh Ramanathan, Dhruv Mahajan

专题命中 GUI与屏幕智能体 ：vision-language model(abstract)；分类 cs.CV

Comments CVPR 2023

Lexi: Self-Supervised Learning of the UI Language

Pratyay Banerjee, Shweti Mahajan, Kushal Arora, Chitta Baral, Oriana Riva

专题命中 GUI与屏幕智能体 ：grounding(abstract)；分类 cs.AI

Comments EMNLP (Findings) 2022