视觉大模型 / VLM - arXivDaily 专题

2305.13669 2024-06-14 cs.CL cs.AI 57%

The Knowledge Alignment Problem: Bridging Human and External Knowledge for Large Language Models

Shuo Zhang, Liangming Pan, Junzhou Zhao, William Yang Wang

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.AI

Comments ACL 2024, Findings

2406.04334 2024-06-07 cs.CV 57%

DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs

Lingchen Meng, Jianwei Yang, Rui Tian, Xiyang Dai, Zuxuan Wu, Jianfeng Gao, Yu-Gang Jiang

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV

Comments Project Page: https://deepstack-vl.github.io/

2406.04292 2024-06-07 cs.IR cs.CL cs.CV 57%

VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval

Junjie Zhou, Zheng Liu, Shitao Xiao, Bo Zhao, Yongping Xiong

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

Comments Accepted to ACL 2024 main conference

2405.20339 2024-05-31 cs.CV 57%

Visual Perception by Large Language Model's Weights

Feipeng Ma, Hongwei Xue, Guangting Wang, Yizhou Zhou, Fengyun Rao, Shilin Yan, Yueyi Zhang, Siying Wu, Mike Zheng Shou, Xiaoyan Sun

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV

2405.18438 2024-05-30 cs.CV 57%

GHOST: Grounded Human Motion Generation with Open Vocabulary Scene-and-Text Contexts

Zoltán Á. Milacski, Koichiro Niinuma, Ryosuke Kawamura, Fernando de la Torre, László A. Jeni

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.CV

Comments 18 pages, 5 figures

2308.04152 2024-05-28 cs.CV 57%

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

Juncheng Li, Kaihang Pan, Zhiqi Ge, Minghe Gao, Wei Ji, Wenqiao Zhang, Tat-Seng Chua, Siliang Tang, Hanwang Zhang, Yueting Zhuang

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV

Comments Accepted by ICLR 2024 (Spotlight)

2405.15738 2024-05-27 cs.CV 57%

ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models

Chunjiang Ge, Sijie Cheng, Ziming Wang, Jiale Yuan, Yuan Gao, Jun Song, Shiji Song, Gao Huang, Bo Zheng

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV

Comments 17 pages

2405.15155 2024-05-27 cs.CV 57%

CLIP model is an Efficient Online Lifelong Learner

Leyuan Wang, Liuyu Xiang, Yujie Wei, Yunlong Wang, Zhaofeng He

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

2405.08748 2024-05-15 cs.CV 57%

Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding

Zhimin Li, Jianwei Zhang, Qin Lin, Jiangfeng Xiong, Yanxin Long, Xinchi Deng, Yingfang Zhang, Xingchao Liu, Minbin Huang, Zedong Xiao, Dayou Chen, Jiajun He, Jiahao Li, Wenyue Li, Chen Zhang, Rongwei Quan, Jianxiang Lu, Jiabin Huang, Xiaoyan Yuan, Xiaoxiao Zheng, Yixuan Li, Jihong Zhang, Chao Zhang, Meng Chen, Jie Liu, Zheng Fang, Weiyan Wang, Jinbao Xue, Yangyu Tao, Jianchen Zhu, Kai Liu, Sihuan Lin, Yifu Sun, Yun Li, Dongdong Wang, Mingtao Chen, Zhichao Hu, Xiao Xiao, Yan Chen, Yuhong Liu, Wei Liu, Di Wang, Yong Yang, Jie Jiang, Qinglin Lu

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV

Comments Project Page: https://dit.hunyuan.tencent.com/

2405.06926 2024-05-14 cs.CV 57%

TAI++: Text as Image for Multi-Label Image Classification by Co-Learning Transferable Prompt

Xiangyu Wu, Qing-Yuan Jiang, Yang Yang, Yi-Feng Wu, Qing-Guo Chen, Jianfeng Lu

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

Comments Accepted for publication at IJCAI 2024; 13 pages; 11 figures

2405.05949 2024-05-10 cs.CV 57%

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts

Jiachen Li, Xinyao Wang, Sijie Zhu, Chia-Wen Kuo, Lu Xu, Fan Chen, Jitesh Jain, Humphrey Shi, Longyin Wen

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV

2307.05222 2024-05-09 cs.CV 57%

Emu: Generative Pretraining in Multimodality

Quan Sun, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Yueze Wang, Hongcheng Gao, Jingjing Liu, Tiejun Huang, Xinlong Wang

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV

Comments Accepted to ICLR 2024. Code and Models: https://github.com/baaivision/Emu

2310.13023 2024-05-08 cs.CL cs.AI 57%

GraphGPT: Graph Instruction Tuning for Large Language Models

Jiabin Tang, Yuhao Yang, Wei Wei, Lei Shi, Lixin Su, Suqi Cheng, Dawei Yin, Chao Huang

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.AI

Comments Accepted by SIGIR'2024, full paper

2405.03194 2024-05-07 cs.CV 57%

CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenario

Zhizhao Duan, Hao Cheng, Duo Xu, Xi Wu, Xiangxie Zhang, Xi Ye, Zhen Xie

专题命中 VLM训练与架构：visual language model(abstract)；分类 cs.CV

Comments Accepted by AICITY2024 Workshop Track2 at CVPR2024

2402.15726 2024-05-07 cs.CV 57%

CLIPose: Category-Level Object Pose Estimation with Pre-trained Vision-Language Knowledge

Xiao Lin, Minghao Zhu, Ronghao Dang, Guangliang Zhou, Shaolong Shu, Feng Lin, Chengju Liu, Qijun Chen

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

Comments 14 pages, 4 figures, 9 tables

2405.00349 2024-05-07 cs.LG 57%

A Self-explaining Neural Architecture for Generalizable Concept Learning

Sanchit Sinha, Guangzhi Xiong, Aidong Zhang

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.LG

Comments IJCAI 2024. 16 pages (7 main content, 2 references, 7 Appendix) Code available at https://github.com/sanchit97/secl

2404.19168 2024-05-01 cs.CV 57%

PEVA-Net: Prompt-Enhanced View Aggregation Network for Zero/Few-Shot Multi-View 3D Shape Recognition

Dongyun Lin, Yi Cheng, Shangbo Mao, Aiyuan Guo, Yiqun Li

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

2404.16365 2024-04-26 cs.CL cs.AI 57%

VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations

Sri Harsha Dumpala, Aman Jaiswal, Chandramouli Sastry, Evangelos Milios, Sageev Oore, Hassan Sajjad

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.AI

2311.15145 2024-04-24 cs.CV 57%

Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalization

Jixuan Leng, Yijiang Li, Haohan Wang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

2404.14412 2024-04-23 cs.CV 57%

AutoAD III: The Prequel -- Back to the Pixels

Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, Andrew Zisserman

专题命中 VLM训练与架构：visual language model(abstract)；分类 cs.CV

Comments CVPR2024. Project page: https://www.robots.ox.ac.uk/~vgg/research/autoad/

2308.12462 2024-04-22 cs.CV 57%

Overcoming Generic Knowledge Loss with Selective Parameter Update

Wenxuan Zhang, Paul Janson, Rahaf Aljundi, Mohamed Elhoseiny

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

2402.06764 2024-04-19 cs.AI 57%

GLaM: Fine-Tuning Large Language Models for Domain Knowledge Graph Alignment via Neighborhood Partitioning and Generative Subgraph Encoding

Stefan Dernbach, Khushbu Agarwal, Alejandro Zuniga, Michael Henry, Sutanay Choudhury

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.AI

Comments Published in AAAI Spring Symposium: AAAI-MAKE 2024

2404.11249 2024-04-18 cs.CV 57%

A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene

Wenbo Zhang, Yifan Zhang, Jianfeng Lin, Binqiang Huang, Jinlu Zhang, Wenhao Yu

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

2404.10146 2024-04-17 cs.CV 57%

Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn Classification without Labels

Amaya Dharmasiri, Muzammal Naseer, Salman Khan, Fahad Shahbaz Khan

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.CV

Comments To be published in Workshop for Learning 3D with Multi-View Supervision (3DMV) at CVPR 2024

2404.08226 2024-04-15 cs.CV 57%

Improving Continuous Sign Language Recognition with Adapted Image Models

Lianyu Hu, Tongkai Shi, Liqing Gao, Zekang Liu, Wei Feng

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

2312.03884 2024-04-15 cs.CV cs.GR 57%

WonderJourney: Going from Anywhere to Everywhere

Hong-Xing Yu, Haoyi Duan, Junhwa Hur, Kyle Sargent, Michael Rubinstein, William T. Freeman, Forrester Cole, Deqing Sun, Noah Snavely, Jiajun Wu, Charles Herrmann

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV

Comments Project website with video results: https://kovenyu.com/WonderJourney/

2402.05195 2024-04-11 cs.CV cs.CL 57%

$λ$-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space

Maitreya Patel, Sangmin Jung, Chitta Baral, Yezhou Yang

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV

Comments Project page: https://eclipse-t2i.github.io/Lambda-ECLIPSE/

2401.06395 2024-04-05 cs.CV 57%

ModaVerse: Efficiently Transforming Modalities with LLMs

Xinyu Wang, Bohan Zhuang, Qi Wu

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV

Comments CVPR2024

2312.01987 2024-04-05 cs.CV 57%

Bootstrapping SparseFormers from Vision Foundation Models

Ziteng Gao, Zhan Tong, Kevin Qinghong Lin, Joya Chen, Mike Zheng Shou

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV

Comments CVPR 2024

2403.17683 2024-04-02 cs.AI 57%

Solution for Emotion Prediction Competition of Workshop on Emotionally and Culturally Intelligent AI

Shengdong Xu, Zhouyang Chi, Yang Yang

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.AI

1. VLM训练与架构 5058 篇

The Knowledge Alignment Problem: Bridging Human and External Knowledge for Large Language Models

DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMs

VISTA: Visualized Text Embedding For Universal Multi-Modal Retrieval

Visual Perception by Large Language Model's Weights

GHOST: Grounded Human Motion Generation with Open Vocabulary Scene-and-Text Contexts

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models

CLIP model is an Efficient Online Lifelong Learner

Hunyuan-DiT: A Powerful Multi-Resolution Diffusion Transformer with Fine-Grained Chinese Understanding

TAI++: Text as Image for Multi-Label Image Classification by Co-Learning Transferable Prompt

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts

Emu: Generative Pretraining in Multimodality

GraphGPT: Graph Instruction Tuning for Large Language Models

CityLLaVA: Efficient Fine-Tuning for VLMs in City Scenario

CLIPose: Category-Level Object Pose Estimation with Pre-trained Vision-Language Knowledge

A Self-explaining Neural Architecture for Generalizable Concept Learning

PEVA-Net: Prompt-Enhanced View Aggregation Network for Zero/Few-Shot Multi-View 3D Shape Recognition

VISLA Benchmark: Evaluating Embedding Sensitivity to Semantic and Lexical Alterations

Choosing Wisely and Learning Deeply: Selective Cross-Modality Distillation via CLIP for Domain Generalization

AutoAD III: The Prequel -- Back to the Pixels

Overcoming Generic Knowledge Loss with Selective Parameter Update

GLaM: Fine-Tuning Large Language Models for Domain Knowledge Graph Alignment via Neighborhood Partitioning and Generative Subgraph Encoding

A Progressive Framework of Vision-language Knowledge Distillation and Alignment for Multilingual Scene

Cross-Modal Self-Training: Aligning Images and Pointclouds to Learn Classification without Labels

Improving Continuous Sign Language Recognition with Adapted Image Models

WonderJourney: Going from Anywhere to Everywhere

$λ$-ECLIPSE: Multi-Concept Personalized Text-to-Image Diffusion Models by Leveraging CLIP Latent Space

ModaVerse: Efficiently Transforming Modalities with LLMs

Bootstrapping SparseFormers from Vision Foundation Models

Solution for Emotion Prediction Competition of Workshop on Emotionally and Culturally Intelligent AI