多模态大模型

A Survey on (M)LLM-Based GUI Agents

Fei Tang, Haolei Xu, Hang Zhang, Siqi Chen, Xingyu Wu, Yongliang Shen, Wenqi Zhang, Guiyang Hou, Zeqi Tan, Yuchen Yan, Kaitao Song, Jian Shao, Weiming Lu, Jun Xiao, Yueting Zhuang

机构 * Zhejiang University（浙江大学） ； Microsoft Research Asia（微软亚洲研究院） 纠错

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

A Survey of LLM-based Agents in Medicine: How far are we from Baymax?

Wenxuan Wang, Zizhan Ma, Zheng Wang, Chenghan Wu, Jiaming Ji, Wenting Chen, Xiang Li, Yixuan Yuan

机构 * The Chinese University of Hong Kong（香港中文大学） ； Peking University（北京大学） ； City University of Hong Kong（香港城市大学） ； Massachusetts General Hospital and Harvard Medical School（麻省总医院和哈佛医学院） 纠错

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2025 Findings

BadNAVer: Exploring Jailbreak Attacks On Vision-and-Language Navigation

Wenqi Lyu, Zerui Li, Yanyuan Qiao, Qi Wu

机构 * Australian Institute for Machine Learning, University of Adelaide（澳大利亚机器学习研究所，阿德莱德大学） 纠错

专题命中 多模态Agent ：multimodal(abstract)；MLLM(abstract)

Comments 8 pages, 4 figures

SceneScout: Towards AI Agent-driven Access to Street View Imagery for Blind Users

Gaurav Jain, Leah Findlater, Cole Gleason

专题命中 多模态Agent ：multimodal(abstract)；MLLM(abstract)

Multi-Agent System for AI-Assisted Extraction of Narrative Arcs in TV Series

Roberto Balestri, Guglielmo Pescatore

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI、cs.MM

Comments 17th International Conference on Agents and Artificial Intelligence, Porto (Portugal). 23/02/2025 - 25/02/2025

Journal ref 2025. Proceedings of the 17th International Conference on Agents and Artificial Intelligence - Volume 1, ISBN 978-989-758-737-5, ISSN 2184-433X, pages 663-670

GUI-Bee: Align GUI Action Grounding to Novel Environments via Autonomous Exploration

Yue Fan, Handong Zhao, Ruiyi Zhang, Yu Shen, Xin Eric Wang, Gang Wu

专题命中 多模态Agent ：MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

PlanLLM: Video Procedure Planning with Refinable Large Language Models

Dejie Yang, Zijing Zhao, Yang Liu

专题命中 多模态Agent ：cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments accepted to AAAI2025

Seeing is Believing: Vision-driven Non-crash Functional Bug Detection for Mobile Apps

Zhe Liu, Cheng Li, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Yawen Wang, Jun Hu, Qing Wang

专题命中 多模态Agent ：multimodal(abstract)；MLLM(abstract)

ADAM: An Embodied Causal Agent in Open-World Environments

Shu Yu, Chaochao Lu

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Agent S: An Open Agentic Framework that Uses Computers Like a Human

Saaket Agashe, Jiuzhou Han, Shuyu Gan, Jiachen Yang, Ang Li, Xin Eric Wang

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 23 pages, 16 figures, 9 tables

"Is This It?": Towards Ecologically Valid Benchmarks for Situated Collaboration

Dan Bohus, Sean Andrist, Yuwei Bao, Eric Horvitz, Ann Paradiso

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI、cs.MM

Towards Human-Level Understanding of Complex Process Engineering Schematics: A Pedagogical, Introspective Multi-Agent Framework for Open-Domain Question Answering

Sagar Srinivas Sakhinana, Geethan Sannidhi, Venkataramana Runkana

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Our paper is accepted for publication at ML4CCE workshop at ECML PKDD 2024

LEGENT: Open Platform for Embodied Agents

Zhili Cheng, Zhitong Wang, Jinyi Hu, Shengding Hu, An Liu, Yuge Tu, Pengkai Li, Lei Shi, Zhiyuan Liu, Maosong Sun

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ACL 2024 System Demonstration

OmniParser for Pure Vision Based GUI Agent

Yadong Lu, Jianwei Yang, Yelong Shen, Ahmed Awadallah

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

LLM-Optic: Unveiling the Capabilities of Large Language Models for Universal Visual Grounding

Haoyu Zhao, Wenhang Ge, Ying-cong Chen

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project Page: https://haoyu-zhao.github.io/LLM-Optic.github.io/

Automatic Layout Planning for Visually-Rich Documents with Instruction-Following Models

Wanrong Zhu, Jennifer Healey, Ruiyi Zhang, William Yang Wang, Tong Sun

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

VideoAgent: Long-form Video Understanding with Large Language Model as Agent

Xiaohan Wang, Yuhui Zhang, Orr Zohar, Serena Yeung-Levy

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

AesopAgent: Agent-driven Evolutionary System on Story-to-Video Production

Jiuniu Wang, Zehua Du, Yuyuan Zhao, Bo Yuan, Kexiang Wang, Jian Liang, Yaxi Zhao, Yihen Lu, Gengliang Li, Junlong Gao, Xin Tu, Zhenyu Guo

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments 22 pages, 13 figures

GPT-4V(ision) is a Generalist Web Agent, if Grounded

Boyuan Zheng, Boyu Gou, Jihyung Kil, Huan Sun, Yu Su

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

SCHEMA: State CHangEs MAtter for Procedure Planning in Instructional Videos

Yulei Niu, Wenliang Guo, Long Chen, Xudong Lin, Shih-Fu Chang

专题命中 多模态Agent ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by ICLR 2024

MultiPLY: A Multisensory Object-Centric Embodied Large Language Model in 3D World

Yining Hong, Zishuo Zheng, Peihao Chen, Yian Wang, Junyan Li, Chuang Gan

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Project page: https://vis-www.cs.umass.edu/multiply

Cosmological parameter estimation with Genetic Algorithms

Ricardo Medel-Esquivel, Isidro Gómez-Vargas, Alejandro A. Morales Sánchez, Ricardo García-Salcedo, J. Alberto Vázquez

专题命中 多模态Agent ：multimodal(abstract)；multi-modal(abstract)

Comments 19 pages, 6 figures; matches version published in Universe

Journal ref Universe 10 (2024), 11

MATK: The Meme Analytical Tool Kit

Ming Shan Hee, Aditi Kumaresan, Nguyen Khoi Hoang, Nirmalendu Prakash, Rui Cao, Roy Ka-Wei Lee

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted at ACM Multimedia'23 Open-Source Software Competition Track

Chameleon: Plug-and-Play Compositional Reasoning with Large Language Models

Pan Lu, Baolin Peng, Hao Cheng, Michel Galley, Kai-Wei Chang, Ying Nian Wu, Song-Chun Zhu, Jianfeng Gao

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 32 pages, 10 figures, 24 tables. Accepted to NeurIPS 2023

Core Challenges in Embodied Vision-Language Planning

Jonathan Francis, Nariaki Kitamura, Felix Labelle, Xiaopeng Lu, Ingrid Navarro, Jean Oh

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Extended Abstract accepted to the 32nd International Joint Conference on Artificial Intelligence (IJCAI 2023); special journal track for authors of published JAIR 2022 and AIJ 2022 papers. 6 pages, 2 figures. arXiv admin note: substantial text overlap with arXiv:2106.13948

Tree-structured Policy Planning with Learned Behavior Models

Yuxiao Chen, Peter Karkus, Boris Ivanovic, Xinshuo Weng, Marco Pavone

专题命中 多模态Agent ：multimodal(abstract)；multi-modal(abstract)

MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge

Linxi Fan, Guanzhi Wang, Yunfan Jiang, Ajay Mandlekar, Yuncong Yang, Haoyi Zhu, Andrew Tang, De-An Huang, Yuke Zhu, Anima Anandkumar

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Outstanding Paper Award at NeurIPS 2022. Project website: https://minedojo.org

ADAPT: Vision-Language Navigation with Modality-Aligned Action Prompts

Bingqian Lin, Yi Zhu, Zicong Chen, Xiwen Liang, Jianzhuang Liu, Xiaodan Liang

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted to CVPR 2022

Core Challenges in Embodied Vision-Language Planning

Jonathan Francis, Nariaki Kitamura, Felix Labelle, Xiaopeng Lu, Ingrid Navarro, Jean Oh

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Journal of Artificial Intelligence Research 74 (2022) 459-515

Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion

Alessandro Suglia, Qiaozi Gao, Jesse Thomason, Govind Thattai, Gaurav Sukhatme

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted at Novel Ideas in Learning-to-Learn through Interaction (NILLI) workshop @ EMNLP 2021