视觉大模型 / VLM - arXivDaily 专题

2411.05939 2024-11-12 cs.CV cs.AI cs.LG 67%

GCI-ViTAL: Gradual Confidence Improvement with Vision Transformers for Active Learning on Label Noise

Moseli Mots'oehli, kyungim Baek

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments under review

2411.05001 2024-11-08 cs.CV cs.AI cs.CL cs.LG 67%

Analyzing The Language of Visual Tokens

David M. Chan, Rodolfo Corona, Joonyong Park, Cheol Jun Cho, Yutong Bai, Trevor Darrell

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

2410.23230 2024-11-01 cs.CV cs.AI cs.LG cs.MM cs.SD eess.AS 67%

Aligning Audio-Visual Joint Representations with an Agentic Workflow

Shentong Mo, Yibing Song

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

2406.18043 2024-11-01 cs.AI cs.CV cs.LG cs.RO 67%

GenRL: Multimodal-foundation world models for generalization in embodied agents

Pietro Mazzaglia, Tim Verbelen, Bart Dhoedt, Aaron Courville, Sai Rajeswar

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Presented at NeurIPS 2024

2406.11833 2024-10-30 cs.CV cs.AI cs.LG 67%

MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs

Ziyu Liu, Tao Chu, Yuhang Zang, Xilin Wei, Xiaoyi Dong, Pan Zhang, Zijian Liang, Yuanjun Xiong, Yu Qiao, Dahua Lin, Jiaqi Wang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments This project is available at https://github.com/Liuziyu77/MMDU

2407.15793 2024-10-29 cs.CV cs.AI cs.LG 67%

CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning

Emanuele Frascaroli, Aniello Panariello, Pietro Buzzega, Lorenzo Bonicelli, Angelo Porrello, Simone Calderara

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 15 pages, 1 figure. Accepted as ORAL at the The 35th British Machine Vision Conference 2024 (BMVC 2024), Glasgow, UK

2406.00093 2024-10-04 cs.CV cs.AI cs.GR cs.LG cs.MM 67%

Bootstrap3D: Improving Multi-view Diffusion Model with Synthetic Data

Zeyi Sun, Tong Wu, Pan Zhang, Yuhang Zang, Xiaoyi Dong, Yuanjun Xiong, Dahua Lin, Jiaqi Wang

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Project Page: https://sunzey.github.io/Bootstrap3D/

2309.07760 2024-09-17 cs.CV cs.AI cs.LG 67%

PRE: Vision-Language Prompt Learning with Reparameterization Encoder

Thi Minh Anh Pham, An Duc Nguyen, Cephas Svosve, Vasileios Argyriou, Georgios Tzimiropoulos

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 10 pages excluding References and Appendix

2408.15305 2024-08-29 cs.LG cs.AI cs.CV 67%

Parameter-Efficient Quantized Mixture-of-Experts Meets Vision-Language Instruction Tuning for Semiconductor Electron Micrograph Analysis

Sakhinana Sagar Srinivas, Chidaksh Ravuru, Geethan Sannidhi, Venkataramana Runkana

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Paper published at ICML 2024 Workshop on Foundation Models in the Wild

2408.04840 2024-08-14 cs.CV cs.AI cs.CL cs.LG 67%

mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models

Jiabo Ye, Haiyang Xu, Haowei Liu, Anwen Hu, Ming Yan, Qi Qian, Ji Zhang, Fei Huang, Jingren Zhou

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV、cs.AI、cs.LG

2405.17430 2024-07-30 cs.CV cs.AI cs.CL cs.LG 67%

Matryoshka Multimodal Models

Mu Cai, Jianwei Yang, Jianfeng Gao, Yong Jae Lee

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Project Page: https://matryoshka-mm.github.io/

2407.00556 2024-07-02 cs.MM 67%

Revisiting Vision-Language Features Adaptation and Inconsistency for Social Media Popularity Prediction

Chih-Chung Hsu, Chia-Ming Lee, Yu-Fan Lin, Yi-Shiuan Chou, Chih-Yu Jian, Chi-Han Tsai

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)

Comments Submission of the 7th Social Media Prediction Challenge

2406.19736 2024-07-01 cs.CV cs.AI cs.CL cs.LG 67%

MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment

Jihao Liu, Xin Huang, Jinliang Zheng, Boxiao Liu, Jia Wang, Osamu Yoshie, Yu Liu, Hongsheng Li

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Dataset and models are available at https://github.com/jihaonew/MM-Instruct

2406.09412 2024-06-14 cs.CV cs.AI cs.LG cs.MM 67%

Explore the Limits of Omni-modal Pretraining at Scale

Yiyuan Zhang, Handong Li, Jing Liu, Xiangyu Yue

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Project Website: https://invictus717.github.io/MiCo/

2406.09388 2024-06-14 cs.CV cs.AI cs.LG 67%

Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition

Youngtaek Oh, Pyunghwan Ahn, Jinhyung Kim, Gwangmo Song, Soonyoung Lee, In So Kweon, Junmo Kim

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted to CVPRW 2024 on 'What is Next in Multimodal Foundation Models?'. Code: https://github.com/ytaek-oh/vl_compo

2406.06465 2024-06-11 cs.CV cs.AI cs.CL cs.LG cs.MM 67%

AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction

Zhen Xing, Qi Dai, Zejia Weng, Zuxuan Wu, Yu-Gang Jiang

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV、cs.AI、cs.LG

2406.00645 2024-06-06 cs.LG cs.AI cs.CV 67%

FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning

Yuwei Fu, Haichao Zhang, Di Wu, Wei Xu, Benoit Boulet

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments ICML 2024

2405.14785 2024-06-05 cs.CV cs.AI cs.LG 67%

EditWorld: Simulating World Dynamics for Instruction-Following Image Editing

Ling Yang, Bohan Zeng, Jiaming Liu, Hong Li, Minghao Xu, Wentao Zhang, Shuicheng Yan

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Project: https://github.com/YangLing0818/EditWorld

2405.04883 2024-05-13 cs.CV cs.AI cs.LG 67%

FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion

Zehan Wang, Ziang Zhang, Xize Cheng, Rongjie Huang, Luping Liu, Zhenhui Ye, Haifeng Huang, Yang Zhao, Tao Jin, Peng Gao, Zhou Zhao

专题命中 VLM训练与架构：InternVL(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments Accepted by ICML 2024. The code and checkpoints will be released at https://github.com/zehanwang01/FreeBind

2310.12973 2024-05-07 cs.CV cs.AI cs.CL cs.LG 67%

Frozen Transformers in Language Models Are Effective Visual Encoder Layers

Ziqi Pang, Ziyang Xie, Yunze Man, Yu-Xiong Wang

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments ICLR 2024 Spotlight. 23 pages, 13 figures. Code at https://github.com/ziqipang/LM4VisualEncoding

2405.01413 2024-05-03 cs.CV cs.AI cs.CL cs.LG 67%

MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors

Yuan Tang, Xu Han, Xianzhi Li, Qiao Yu, Yixue Hao, Long Hu, Min Chen

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 17 pages, 9 figures

2404.19753 2024-05-01 cs.CV cs.AI cs.CL cs.LG 67%

DOCCI: Descriptions of Connected and Contrasting Images

Yasumasa Onoe, Sunayana Rane, Zachary Berger, Yonatan Bitton, Jaemin Cho, Roopal Garg, Alexander Ku, Zarana Parekh, Jordi Pont-Tuset, Garrett Tanzer, Su Wang, Jason Baldridge

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

2404.06212 2024-04-10 cs.CV cs.AI cs.LG 67%

OmniFusion Technical Report

Elizaveta Goncharova, Anton Razzhigaev, Matvey Mikhalchuk, Maxim Kurkin, Irina Abdullaeva, Matvey Skripkin, Ivan Oseledets, Denis Dimitrov, Andrey Kuznetsov

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 17 pages, 4 figures, 9 tables, 2 appendices

2310.07699 2024-03-15 cs.CV cs.AI cs.LG 67%

VeCLIP: Improving CLIP Training via Visual-enriched Captions

Zhengfeng Lai, Haotian Zhang, Bowen Zhang, Wentao Wu, Haoping Bai, Aleksei Timofeev, Xianzhi Du, Zhe Gan, Jiulong Shan, Chen-Nee Chuah, Yinfei Yang, Meng Cao

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments CV/ML

2312.07424 2024-02-27 cs.LG cs.AI cs.CV 67%

How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigation

Zhongyi Han, Guanglin Zhou, Rundong He, Jindong Wang, Tailin Wu, Yilong Yin, Salman Khan, Lina Yao, Tongliang Liu, Kun Zhang

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments added the investigation of Gemini. 66 pages, 41 figures

2402.12509 2024-02-21 cs.RO 67%

Talk Through It: End User Directed Manipulation Learning

Carl Winge, Adam Imdieke, Bahaa Aldeeb, Dongyeop Kang, Karthik Desingh

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)

2309.01516 2024-02-07 cs.CV cs.AI cs.LG cs.MM 67%

MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval

Zijun Long, George Killick, Richard McCreadie, Gerardo Aragon Camarasa

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV、cs.AI、cs.LG

2401.11206 2024-01-23 cs.CL 67%

InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance

Pengyu Wang, Dong Zhang, Linyang Li, Chenkun Tan, Xinghao Wang, Ke Ren, Botian Jiang, Xipeng Qiu

专题命中 VLM训练与架构：LLaVA(abstract)；multimodal large language model(abstract)

2401.09865 2024-01-19 cs.CV cs.AI cs.LG 67%

Improving fine-grained understanding in image-text pre-training

Ioana Bica, Anastasija Ilić, Matthias Bauer, Goker Erdogan, Matko Bošnjak, Christos Kaplanis, Alexey A. Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, Jovana Mitrović

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 26 pages

2305.13301 2024-01-08 cs.LG cs.AI cs.CV 67%

Training Diffusion Models with Reinforcement Learning

Kevin Black, Michael Janner, Yilun Du, Ilya Kostrikov, Sergey Levine

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI、cs.LG

Comments 23 pages, 16 figures

1. VLM训练与架构 5058 篇

GCI-ViTAL: Gradual Confidence Improvement with Vision Transformers for Active Learning on Label Noise

Analyzing The Language of Visual Tokens

Aligning Audio-Visual Joint Representations with an Agentic Workflow

GenRL: Multimodal-foundation world models for generalization in embodied agents

MMDU: A Multi-Turn Multi-Image Dialog Understanding Benchmark and Instruction-Tuning Dataset for LVLMs

CLIP with Generative Latent Replay: a Strong Baseline for Incremental Learning

Bootstrap3D: Improving Multi-view Diffusion Model with Synthetic Data

PRE: Vision-Language Prompt Learning with Reparameterization Encoder

Parameter-Efficient Quantized Mixture-of-Experts Meets Vision-Language Instruction Tuning for Semiconductor Electron Micrograph Analysis

mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models

Matryoshka Multimodal Models

Revisiting Vision-Language Features Adaptation and Inconsistency for Social Media Popularity Prediction

MM-Instruct: Generated Visual Instructions for Large Multimodal Model Alignment

Explore the Limits of Omni-modal Pretraining at Scale

Exploring the Spectrum of Visio-Linguistic Compositionality and Recognition

AID: Adapting Image2Video Diffusion Models for Instruction-guided Video Prediction

FuRL: Visual-Language Models as Fuzzy Rewards for Reinforcement Learning

EditWorld: Simulating World Dynamics for Instruction-Following Image Editing

FreeBind: Free Lunch in Unified Multimodal Space via Knowledge Fusion

Frozen Transformers in Language Models Are Effective Visual Encoder Layers

MiniGPT-3D: Efficiently Aligning 3D Point Clouds with Large Language Models using 2D Priors

DOCCI: Descriptions of Connected and Contrasting Images

OmniFusion Technical Report

VeCLIP: Improving CLIP Training via Visual-enriched Captions

How Well Does GPT-4V(ision) Adapt to Distribution Shifts? A Preliminary Investigation

Talk Through It: End User Directed Manipulation Learning

MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval

InferAligner: Inference-Time Alignment for Harmlessness through Cross-Model Guidance

Improving fine-grained understanding in image-text pre-training

Training Diffusion Models with Reinforcement Learning