多模态大模型

Surfer: Progressive Reasoning with World Models for Robotic Manipulation

Pengzhen Ren, Kaidong Zhang, Hetao Zheng, Zixuan Li, Yuhang Wen, Fengda Zhu, Mas Ma, Xiaodan Liang

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.AI

ByteComposer: a Human-like Melody Composition Method based on Language Model Agent

Xia Liang, Xingjian Du, Jiaju Lin, Pei Zou, Yuan Wan, Bilei Zhu

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.AI、eess.AS

Pragmatic Instruction Following and Goal Assistance via Cooperative Language-Guided Inverse Planning

Tan Zhi-Xuan, Lance Ying, Vikash Mansinghka, Joshua B. Tenenbaum

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Accepted to AAMAS 2024. 8 pages (excl. references), 5 figures/tables. (Appendix: 8 pages, 8 figures/tables). Code available at: https://github.com/probcomp/CLIPS.jl

ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments

Dong An, Hanqing Wang, Wenguan Wang, Zun Wang, Yan Huang, Keji He, Liang Wang

专题命中 多模态Agent ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Project page: https://github.com/MarSaKi/ETPNav

Morphological Profiling for Drug Discovery in the Era of Deep Learning

Qiaosi Tang, Ranjala Ratnayake, Gustavo Seabra, Zhe Jiang, Ruogu Fang, Lina Cui, Yousong Ding, Tamer Kahveci, Jiang Bian, Chenglong Li, Hendrik Luesch, Yanjun Li

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 44 pages, 5 figure, 5 tables

LACMA: Language-Aligning Contrastive Learning with Meta-Actions for Embodied Instruction Following

Cheng-Fu Yang, Yen-Chun Chen, Jianwei Yang, Xiyang Dai, Lu Yuan, Yu-Chiang Frank Wang, Kai-Wei Chang

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments EMNLP 2023

MotionLM: Multi-Agent Motion Forecasting as Language Modeling

Ari Seff, Brian Cera, Dian Chen, Mason Ng, Aurick Zhou, Nigamaa Nayakanti, Khaled S. Refaat, Rami Al-Rfou, Benjamin Sapp

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments To appear at the International Conference on Computer Vision (ICCV) 2023

Rating Prediction in Conversational Task Assistants with Behavioral and Conversational-Flow Features

Rafael Ferreira, David Semedo, João Magalhães

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought

Yao Mu, Qinglong Zhang, Mengkang Hu, Wenhai Wang, Mingyu Ding, Jun Jin, Bin Wang, Jifeng Dai, Yu Qiao, Ping Luo

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.AI

Sequential annotations for naturally-occurring HRI: first insights

Lucien Tisserand, Frédéric Armetta, Heike Baldauf-Quilliatre, Antoine Bouquin, Salima Hassas, Mathieu Lefort

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Peer-reviewed workshop paper accepted for the ''Human-Robot Conversational Interaction'' workshop that took place at the ''ACM/IEEE International Conference on Human-Robot Interaction'' 2023 Conference in Stockholm, Sweden

Integrated Image and Location Analysis for Wound Classification: A Deep Learning Approach

Yash Patel, Tirth Shah, Mrinal Kanti Dhar, Taiyu Zhang, Jeffrey Niezgoda, Sandeep Gopalakrishnan, Zeyun Yu

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.AI

Inferring the Goals of Communicating Agents from Actions and Instructions

Lance Ying, Tan Zhi-Xuan, Vikash Mansinghka, Joshua B. Tenenbaum

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments 8 pages, 5 figures. Accepted to the ICML 2023 Workshop on Theory of Mind in Communicating Agents. Supplementary Information: https://osf.io/gh758/

"You might think about slightly revising the title": identifying hedges in peer-tutoring interactions

Yann Raphalen, Chloé Clavel, Justine Cassell

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Published in Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), 2022

Journal ref Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), Volume 1: long papers (2022)

LLM as A Robotic Brain: Unifying Egocentric Memory and Control

Jinjie Mai, Jun Chen, Bing Li, Guocheng Qian, Mohamed Elhoseiny, Bernard Ghanem

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments This early project is now integrated to: Mindstorms in Natural Language-Based Societies of Mind, arXiv:2305.17066

Accessible Instruction-Following Agent

Kairui Zhou

专题命中 多模态Agent ：cross-modal(abstract)；分类 cs.CL、cs.AI

EC^2: Emergent Communication for Embodied Control

Yao Mu, Shunyu Yao, Mingyu Ding, Ping Luo, Chuang Gan

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Published in CVPR2023

3D Object Detection for Autonomous Driving: A Comprehensive Survey

Jiageng Mao, Shaoshuai Shi, Xiaogang Wang, Hongsheng Li

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted to International Journal of Computer Vision (IJCV). Project page is at https://github.com/PointsCoder/Awesome-3D-Object-Detection-for-Autonomous-Driving

POTATO: The Portable Text Annotation Tool

Jiaxin Pei, Aparna Ananthasubramaniam, Xingyao Wang, Naitian Zhou, Jackson Sargent, Apostolos Dedeloudis, David Jurgens

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments EMNLP 2022 DEMO

Symbol Emergence as Inter-personal Categorization with Head-to-head Latent Word

Kazuma Furukawa, Akira Taniguchi, Yoshinobu Hagiwara, Tadahiro Taniguchi

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments 7 pages, 4 figures, 5 tables

Journal ref IEEE International Conference on Development and Learning (ICDL 2022), 2022, 60-67

Optimization of Image Transmission in a Cooperative Semantic Communication Networks

Wenjing Zhang, Yining Wang, Mingzhe Chen, Tao Luo, Dusit Niyato

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 29 pages, 10 figures

Distribution-aware Goal Prediction and Conformant Model-based Planning for Safe Autonomous Driving

Jonathan Francis, Bingqing Chen, Weiran Yao, Eric Nyberg, Jean Oh

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted: 1st Workshop on Safe Learning for Autonomous Driving, at the International Conference on Machine Learning (ICML 2022); Best Paper Award

A Generalist Agent

Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, Nando de Freitas

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Published at TMLR, 42 pages

Journal ref Transactions on Machine Learning Research, 11/2022, https://openreview.net/forum?id=1ikK0kHjvj

ERNIE-Layout: Layout Knowledge Enhanced Pre-training for Visually-rich Document Understanding

Qiming Peng, Yinxu Pan, Wenjin Wang, Bin Luo, Zhenyu Zhang, Zhengjie Huang, Teng Hu, Weichong Yin, Yongfeng Chen, Yin Zhang, Shikun Feng, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

专题命中 多模态Agent ：multi-modal(abstract)；分类 cs.CL、cs.AI

Comments Accepted to EMNLP 2022 (Findings)

Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation

Yicong Hong, Zun Wang, Qi Wu, Stephen Gould

专题命中 多模态Agent ：cross-modal(abstract)；分类 cs.CV、cs.CL

Multi-Agent Variational Occlusion Inference Using People as Sensors

Masha Itkina, Ye-Ji Mun, Katherine Driggs-Campbell, Mykel J. Kochenderfer

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments 12 pages, 9 figures, International Conference on Robotics and Automation (ICRA) 2022

Building Goal-Oriented Dialogue Systems with Situated Visual Context

Sanchit Agarwal, Jan Jezabek, Arijit Biswas, Emre Barut, Shuyang Gao, Tagyoung Chung

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.CL

SocialAI: Benchmarking Socio-Cognitive Abilities in Deep Reinforcement Learning Agents

Grgur Kovač, Rémy Portelas, Katja Hofmann, Pierre-Yves Oudeyer

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments under review. This paper extends and generalizes work in arXiv:2104.13207

Communicative Learning with Natural Gestures for Embodied Navigation Agents with Human-in-the-Scene

Qi Wu, Cheng-Ju Wu, Yixin Zhu, Jungseock Joo

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments To appear in IROS 2021

Maria: A Visual Experience Powered Conversational Agent

Zujie Liang, Huang Hu, Can Xu, Chongyang Tao, Xiubo Geng, Yining Chen, Fan Liang, Daxin Jiang

专题命中 多模态Agent ：multimodal(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2021 main conference

Towards Distraction-Robust Active Visual Tracking

Fangwei Zhong, Peng Sun, Wenhan Luo, Tingyun Yan, Yizhou Wang

专题命中 多模态Agent ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments To appear in ICML2021