视觉大模型 / VLM - arXivDaily 专题

2303.12513 2023-11-07 cs.CV cs.CL cs.LG 62%

Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding

Morris Alper, Michael Fiman, Hadar Averbuch-Elor

专题命中 VLM训练与架构：visual reasoning(abstract)；分类 cs.CV、cs.LG

Comments Accepted to CVPR 2023. Project webpage: https://isbertblind.github.io/

2311.01064 2023-11-03 cs.CV cs.LG 62%

Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images

Zalan Fabian, Zhongqi Miao, Chunyuan Li, Yuanhan Zhang, Ziwei Liu, Andrés Hernández, Andrés Montes-Rojas, Rafael Escucha, Laura Siabatto, Andrés Link, Pablo Arbeláez, Rahul Dodhia, Juan Lavista Ferres

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments 18 pages, 9 figures

2310.16148 2023-10-26 cs.CV cs.AI 62%

Yin Yang Convolutional Nets: Image Manifold Extraction by the Analysis of Opposites

Augusto Seben da Rosa, Frederico Santos de Oliveira, Anderson da Silva Soares, Arnaldo Candido Junior

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.CV、cs.AI

Comments 12 pages, 5 tables and 6 figures

2310.15145 2023-10-24 cs.RO cs.AI cs.LG 62%

Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning

Jingyun Yang, Max Sobol Mark, Brandon Vu, Archit Sharma, Jeannette Bohg, Chelsea Finn

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.AI、cs.LG

2306.13856 2023-10-24 cs.CV cs.AI 62%

Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification

Rui Wang, Peipei Li, Huaibo Huang, Chunshui Cao, Ran He, Zhaofeng He

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by NeurIPS 2023

2211.15398 2023-09-06 cs.CV cs.LG 62%

Leveraging per Image-Token Consistency for Vision-Language Pre-training

Yunhao Gou, Tom Ko, Hansi Yang, James Kwok, Yu Zhang, Mingxuan Wang

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV、cs.LG

Comments Accepted by CVPR 2023

2308.09970 2023-08-22 cs.CL cs.AI cs.LG 62%

Tackling Vision Language Tasks Through Learning Inner Monologues

Diji Yang, Kezhen Chen, Jinmeng Rao, Xiaoyuan Guo, Yawen Zhang, Jie Yang, Yi Zhang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.AI、cs.LG

2307.07397 2023-07-17 cs.CV cs.LG 62%

Improving Zero-Shot Generalization for CLIP with Synthesized Prompts

Zhengbo Wang, Jian Liang, Ran He, Nan Xu, Zilei Wang, Tieniu Tan

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted by ICCV 2023

2212.10549 2023-07-06 cs.CL cs.CV cs.LG 62%

Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment

Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments ACL 2023

2305.04160 2023-05-23 cs.CL cs.AI cs.CV eess.AS 62%

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

Feilong Chen, Minglun Han, Haozhi Zhao, Qingyang Zhang, Jing Shi, Shuang Xu, Bo Xu

专题命中 VLM训练与架构：visual language model(abstract)；分类 cs.CV、cs.AI

2305.01239 2023-05-03 cs.CV cs.AI 62%

DRPT: Disentangled and Recurrent Prompt Tuning for Compositional Zero-Shot Learning

Xiaocheng Lu, Ziming Liu, Song Guo, Jingcai Guo, Fushuo Huo, Sikai Bai, Tao Han

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2304.14204 2023-04-28 cs.AI cs.CV 62%

Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining

Bingqian Lin, Zicong Chen, Mingjie Li, Haokun Lin, Hang Xu, Yi Zhu, Jianzhuang Liu, Wenjia Cai, Lei Yang, Shen Zhao, Chenfei Wu, Ling Chen, Xiaojun Chang, Yi Yang, Lei Xing, Xiaodan Liang

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.AI

Comments Project page: https://github.com/chenzcv7/MOTOR

2204.03574 2023-04-25 cs.LG cs.CL cs.CV 62%

Learning to Compose Soft Prompts for Compositional Zero-Shot Learning

Nihal V. Nayak, Peilin Yu, Stephen H. Bach

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments ICLR 2023

2208.05140 2023-04-13 eess.IV cs.CL cs.CV cs.LG 62%

Self-supervised Multi-modal Training from Uncurated Image and Reports Enables Zero-shot Oversight Artificial Intelligence in Radiology

Sangjoon Park, Eun Sun Lee, Kyung Sook Shin, Jeong Eun Lee, Jong Chul Ye

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

2303.02995 2023-03-07 cs.CV cs.CL cs.LG 62%

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Shijie Geng, Jianbo Yuan, Yu Tian, Yuxiao Chen, Yongfeng Zhang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted at ICLR 2023

2203.13366 2023-01-04 cs.IR cs.AI cs.CL cs.LG 62%

Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)

Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, Yongfeng Zhang

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.AI、cs.LG

Comments Accepted to ACM RecSys 2022. The arXiv version is more comprehensive

2111.10974 2022-12-29 cs.CV cs.AI cs.CL 62%

Many Heads but One Brain: Fusion Brain -- a Competition and a Single Multimodal Multitask Architecture

Daria Bakshandaeva, Denis Dimitrov, Vladimir Arkhipkin, Alex Shonenkov, Mark Potanin, Denis Karachev, Andrey Kuznetsov, Anton Voronov, Vera Davydova, Elena Tutubalina, Aleksandr Petiushko

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.AI

2211.08025 2022-11-16 cs.LG cs.CL cs.CV 62%

FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers

Jinyu Chen, Wenchao Xu, Song Guo, Junxiao Wang, Jie Zhang, Haozhao Wang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

2201.12596 2022-09-15 cs.CV cs.AI cs.MM 62%

MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning

Zejun Li, Zhihao Fan, Huaixiao Tou, Jingjing Chen, Zhongyu Wei, Xuanjing Huang

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ACM MM22

2105.12210 2021-06-01 cs.LG cs.CV cs.NE 62%

The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design

George Philipp

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.CV、cs.LG

Comments This work is based on the PhD thesis with the same name, author, year and institution. Both works may be cited interchangeably

2103.15679 2021-03-30 cs.CV cs.LG 62%

Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers

Hila Chefer, Shir Gur, Lior Wolf

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.LG

2011.09530 2020-11-20 cs.CV cs.AI eess.IV 62%

Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language

Hassan Akbari, Hamid Palangi, Jianwei Yang, Sudha Rao, Asli Celikyilmaz, Roland Fernandez, Paul Smolensky, Jianfeng Gao, Shih-Fu Chang

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.CV、cs.AI

2006.14744 2020-07-28 cs.CL cs.CV cs.LG 62%

Graph Optimal Transport for Cross-Domain Alignment

Liqun Chen, Zhe Gan, Yu Cheng, Linjie Li, Lawrence Carin, Jingjing Liu

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.LG

Journal ref ICML 2020

1907.07804 2020-07-06 cs.LG cs.CL cs.CV cs.NE stat.ML 62%

OmniNet: A unified architecture for multi-modal multi-task learning

Subhojeet Pramanik, Priyanka Agrawal, Aman Hussain

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.LG

Comments Source code available at: https://github.com/subho406/OmniNet

1912.02379 2020-04-01 cs.LG cs.CL cs.CV stat.ML 62%

Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline

Vishvak Murahari, Dhruv Batra, Devi Parikh, Abhishek Das

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.LG

1908.07490 2019-12-05 cs.CL cs.CV cs.LG 62%

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

Hao Tan, Mohit Bansal

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.LG

Comments EMNLP 2019 (14 pages; with new attention visualizations)

2407.08101 2026-04-16 cs.CV 61%

What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction

说什么和何时说：Live Fitness Coaching作为情境交互的测试平台

Sunny Panchal, Apratim Bhattacharyya, Guillaume Berger, Antoine Mercier, Cornelius Bohm, Florian Dietrichkeit, Reza Pourreza, Xuanlin Li, Pulkit Madan, Mingu Lee, Mark Todorovich, Ingo Bax, Roland Memisevic

机构 * TwentyBN GmbH（TwentyBN公司）； Qualcomm AI Research（高通人工智能研究）； Aignostics GmbH（Aignostics公司）； UC San Diego（加州大学圣地亚哥分校）

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV；VLM(comments)

AI总结本文提出QEVD基准和数据集，研究人类与AI在健身指导中的情境交互，测试视觉语言模型在实时反馈中的能力，揭示现有模型的局限并提出异步流式基线方法。

Comments Accepted to the 2024 NeurIPS Datasets and Benchmarks track; Data: https://www.qualcomm.com/developer/software/qevd-dataset Dataset quick start guide: https://github.com/varworkshop/ai_coach_fitness_2026 and Stream-VLM code: https://github.com/Qualcomm-AI-research/FitCoach

2410.13694 2024-10-18 cs.CV cs.CL 61%

Exploring the Design Space of Visual Context Representation in Video MLLMs

Yifan Du, Yuqi Huo, Kun Zhou, Zijia Zhao, Haoyu Lu, Han Huang, Wayne Xin Zhao, Bingning Wang, Weipeng Chen, Ji-Rong Wen

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV；MLLM(comments)

Comments Long Video MLLM; work in progress

2312.00438 2023-12-04 cs.CV 61%

Dolphins: Multimodal Language Model for Driving

Yingzi Ma, Yulong Cao, Jiachen Sun, Marco Pavone, Chaowei Xiao

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV；VLM(comments)

Comments The project page is available at https://vlm-driver.github.io/

2306.06354 2023-11-20 cs.CV 61%

EventCLIP: Adapting CLIP for Event-based Object Recognition

Ziyi Wu, Xudong Liu, Igor Gilitschenski

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV；VLM(comments)

Comments Add results on 1) EventCLIP with another VLM FLIP 2) inference speed analysis

1. VLM训练与架构 5058 篇

Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding

Multimodal Foundation Models for Zero-shot Animal Species Recognition in Camera Trap Images

Yin Yang Convolutional Nets: Image Manifold Extraction by the Analysis of Opposites

Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learning

Learning-to-Rank Meets Language: Boosting Language-Driven Ordering Alignment for Ordinal Classification

Leveraging per Image-Token Consistency for Vision-Language Pre-training

Tackling Vision Language Tasks Through Learning Inner Monologues

Improving Zero-Shot Generalization for CLIP with Synthesized Prompts

Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment

X-LLM: Bootstrapping Advanced Large Language Models by Treating Multi-Modalities as Foreign Languages

DRPT: Disentangled and Recurrent Prompt Tuning for Compositional Zero-Shot Learning

Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretraining

Learning to Compose Soft Prompts for Compositional Zero-Shot Learning

Self-supervised Multi-modal Training from Uncurated Image and Reports Enables Zero-shot Oversight Artificial Intelligence in Radiology

HiCLIP: Contrastive Language-Image Pretraining with Hierarchy-aware Attention

Recommendation as Language Processing (RLP): A Unified Pretrain, Personalized Prompt & Predict Paradigm (P5)

Many Heads but One Brain: Fusion Brain -- a Competition and a Single Multimodal Multitask Architecture

FedTune: A Deep Dive into Efficient Federated Fine-Tuning with Pre-trained Transformers

MVPTR: Multi-Level Semantic Alignment for Vision-Language Pre-Training via Multi-Stage Learning

The Nonlinearity Coefficient - A Practical Guide to Neural Architecture Design

Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformers

Neuro-Symbolic Representations for Video Captioning: A Case for Leveraging Inductive Biases for Vision and Language

Graph Optimal Transport for Cross-Domain Alignment

OmniNet: A unified architecture for multi-modal multi-task learning

Large-scale Pretraining for Visual Dialog: A Simple State-of-the-Art Baseline

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

What to Say and When to Say it: Live Fitness Coaching as a Testbed for Situated Interaction

Exploring the Design Space of Visual Context Representation in Video MLLMs

Dolphins: Multimodal Language Model for Driving

EventCLIP: Adapting CLIP for Event-based Object Recognition