视觉大模型 / VLM

Compound Expression Recognition via Large Vision-Language Models

Jun Yu, Xilong Lu

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

FedMLLM: Federated Fine-tuning MLLM on Multimodal Heterogeneity Data

Binqian Xu, Xiangbo Shu, Haiyang Mei, Guosen Xie, Basura Fernando, Jinhui Tang

专题命中 VLM训练与架构 ：MLLM(title)；multimodal large language model(abstract)；分类 cs.CV、cs.LG

MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Models

Peng Xia, Kangyu Zhu, Haoran Li, Tianze Wang, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, Huaxiu Yao

专题命中 VLM训练与架构 ：vision language model(title)；vision-language model(abstract)；分类 cs.CV、cs.LG

Comments ICLR 2025

TLDR: Token-Level Detective Reward Model for Large Vision Language Models

Deqing Fu, Tong Xiao, Rui Wang, Wang Zhu, Pengchuan Zhang, Guan Pang, Robin Jia, Lawrence Chen

专题命中 VLM训练与架构 ：vision language model(title)；multimodal large language model(abstract)；分类 cs.CV、cs.LG

Comments Published as a conference paper at ICLR 2025

LongWriter-V: Enabling Ultra-Long and High-Fidelity Generation in Vision-Language Models

Shangqing Tu, Yucheng Wang, Daniel Zhang-Li, Yushi Bai, Jifan Yu, Yuhao Wu, Lei Hou, Huiqin Liu, Zhiyuan Liu, Bin Xu, Juanzi Li

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

PLPHP: Per-Layer Per-Head Vision Token Pruning for Efficient Large Vision-Language Models

Yu Meng, Kaiyuan Li, Chenran Huang, Chen Gao, Xinlei Chen, Yong Li, Xiaoping Zhang

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments 12 pages, 8 figures

CTR-Driven Advertising Image Generation with Multimodal Large Language Models

Xingye Chen, Wei Feng, Zhenbang Du, Weizhen Wang, Yanyin Chen, Haohan Wang, Linkai Liu, Yaoyu Li, Jinyuan Zhao, Yu Li, Zheng Zhang, Jingjing Lv, Junjie Shen, Zhangang Lin, Jingping Shao, Yuanjie Shao, Xinge You, Changxin Gao, Nong Sang

专题命中 VLM训练与架构 ：multimodal large language model(title,abstract)；分类 cs.CV、cs.LG

Comments Accepted to WWW 2025

Linear Alignment of Vision-language Models for Image Captioning

Fabian Paischer, Markus Hofmarcher, Sepp Hochreiter, Thomas Adler

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

Comments 9 pages (+ references and appendix)

Dynamic Token Reduction during Generation for Vision Language Models

Xiaoyu Liang, Chaofeng Guan, Jiaying Lu, Huiyao Chen, Huan Wang, Haoji Hu

专题命中 VLM训练与架构 ：vision language model(title)；vision-language model(abstract)；分类 cs.CV、cs.AI

Instruction-Guided Fusion of Multi-Layer Visual Features in Large Vision-Language Models

Xu Li, Yi Zheng, Haotian Chen, Xiaolei Chen, Yuxuan Liang, Chenghang Lai, Bin Li, Xiangyang Xue

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

BMIP: Bi-directional Modality Interaction Prompt Learning for VLM

Song-Lin Lv, Yu-Yang Chen, Zhi Zhou, Ming Yang, Lan-Zhe Guo

专题命中 VLM训练与架构 ：VLM(title)；vision-language model(abstract)；分类 cs.CV、cs.LG

HyperCLIP: Adapting Vision-Language models with Hypernetworks

Victor Akinwande, Mohammad Sadegh Norouzzadeh, Devin Willmott, Anna Bair, Madan Ravi Ganesh, J. Zico Kolter

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning

Shihao Xu, Yiyang Luo, Wei Shi

专题命中 VLM训练与架构 ：LLaVA(title,abstract)；分类 cs.CV、cs.AI

Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion

Jiuhai Chen, Jianwei Yang, Haiping Wu, Dianqi Li, Jianfeng Gao, Tianyi Zhou, Bin Xiao

专题命中 VLM训练与架构 ：vision-language model(title)；multimodal large language model(abstract)；分类 cs.CV、cs.AI

Molmo and PixMo: Open Weights and Open Data for State-of-the-Art Vision-Language Models

Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, Jiasen Lu, Taira Anderson, Erin Bransom, Kiana Ehsani, Huong Ngo, YenSung Chen, Ajay Patel, Mark Yatskar, Chris Callison-Burch, Andrew Head, Rose Hendrix, Favyen Bastani, Eli VanderBilt, Nathan Lambert, Yvonne Chou, Arnavi Chheda, Jenna Sparks, Sam Skjonsberg, Michael Schmitz, Aaron Sarnat, Byron Bischoff, Pete Walsh, Chris Newell, Piper Wolters, Tanmay Gupta, Kuo-Hao Zeng, Jon Borchardt, Dirk Groeneveld, Crystal Nam, Sophie Lebrecht, Caitlin Wittlif, Carissa Schoenick, Oscar Michel, Ranjay Krishna, Luca Weihs, Noah A. Smith, Hannaneh Hajishirzi, Ross Girshick, Ali Farhadi, Aniruddha Kembhavi

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

Comments Updated with ablations and more technical details

Personalized Multimodal Large Language Models: A Survey

Junda Wu, Hanjia Lyu, Yu Xia, Zhehao Zhang, Joe Barrow, Ishita Kumar, Mehrnoosh Mirtaheri, Hongjie Chen, Ryan A. Rossi, Franck Dernoncourt, Tong Yu, Ruiyi Zhang, Jiuxiang Gu, Nesreen K. Ahmed, Yu Wang, Xiang Chen, Hanieh Deilamsalehy, Namyong Park, Sungchul Kim, Huanrui Yang, Subrata Mitra, Zhengmian Hu, Nedim Lipka, Dang Nguyen, Yue Zhao, Jiebo Luo, Julian McAuley

专题命中 VLM训练与架构 ：multimodal large language model(title,abstract)；分类 cs.CV、cs.AI

FoPru: Focal Pruning for Efficient Large Vision-Language Models

Lei Jiang, Weizhe Huang, Tongxuan Liu, Yuting Zeng, Jing Li, Lechao Cheng, Xiaohua Xu

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments 11 pages, 7 figures

Towards Calibrated Robust Fine-Tuning of Vision-Language Models

Changdae Oh, Hyesu Lim, Mijoo Kim, Dongyoon Han, Sangdoo Yun, Jaegul Choo, Alexander Hauptmann, Zhi-Qi Cheng, Kyungwoo Song

专题命中 VLM训练与架构 ：vision-language model(title)；vision language model(abstract)；分类 cs.CV、cs.AI

Comments NeurIPS 2024 (a short version was presented at the NeurIPS 2023 Workshop on Distribution Shifts); Major modification of (v7): Fixing the x-axis of Figure 3 and Pearson correlation, accordingly

POINTS: Improving Your Vision-language Model with Affordable Strategies

Yuan Liu, Zhongyin Zhao, Ziyuan Zhuang, Le Tian, Xiao Zhou, Jie Zhou

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments v2

Frustratingly Easy Test-Time Adaptation of Vision-Language Models

Matteo Farina, Gianni Franchi, Giovanni Iacca, Massimiliano Mancini, Elisa Ricci

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments Camera-ready version for NeurIPS 2024

Calibrated Self-Rewarding Vision Language Models

Yiyang Zhou, Zhiyuan Fan, Dongjie Cheng, Sihan Yang, Zhaorun Chen, Chenhang Cui, Xiyao Wang, Yun Li, Linjun Zhang, Huaxiu Yao

专题命中 VLM训练与架构 ：vision language model(title)；vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Added some experiments and charts, and redrew some figures V4

LLaVA-VSD: Large Language-and-Vision Assistant for Visual Spatial Description

Yizhang Jin, Jian Li, Jiangning Zhang, Jianlong Hu, Zhenye Gan, Xin Tan, Yong Liu, Yabiao Wang, Chengjie Wang, Lizhuang Ma

专题命中 VLM训练与架构 ：LLaVA(title,abstract)；分类 cs.CV、cs.AI

Comments We have discovered a significant error in the paper that affects the main conclusions. To ensure the accuracy of our research, we have decided to withdraw this paper and will resubmit it after making the necessary corrections

No Filter: Cultural and Socioeconomic Diversity in Contrastive Vision-Language Models

Angéline Pouget, Lucas Beyer, Emanuele Bugliarello, Xiao Wang, Andreas Peter Steiner, Xiaohua Zhai, Ibrahim Alabdulmohsin

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments 17 pages, 5 figures, 4 tables. 38th Conference on Neural Information Processing Systems (NeurIPS 2024)

Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution

Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, Junyang Lin

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

Comments Code is available at https://github.com/QwenLM/Qwen2-VL. arXiv admin note: text overlap with arXiv:2408.15262 by other authors

Vision-Language Models are Strong Noisy Label Detectors

Tong Wei, Hao-Tian Li, Chun-Shu Li, Jiang-Xin Shi, Yu-Feng Li, Min-Ling Zhang

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

Comments Accepted at NeurIPS 2024

A Survey of Multimodal Large Language Model from A Data-centric Perspective

Tianyi Bai, Hao Liang, Binwang Wan, Yanran Xu, Xi Li, Shiyu Li, Ling Yang, Bozhou Li, Yifan Wang, Bin Cui, Ping Huang, Jiulong Shan, Conghui He, Binhang Yuan, Wentao Zhang

专题命中 VLM训练与架构 ：multimodal large language model(title,abstract)；分类 cs.CV、cs.AI

Pre-Trained Vision-Language Models as Partial Annotators

Qian-Wei Wang, Yuqiu Xie, Letian Zhang, Zimo Liu, Shu-Tao Xia

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

RL-VLM-F: Reinforcement Learning from Vision Language Foundation Model Feedback

Yufei Wang, Zhanyi Sun, Jesse Zhang, Zhou Xian, Erdem Biyik, David Held, Zackory Erickson

专题命中 VLM训练与架构 ：VLM(title,abstract)；分类 cs.AI、cs.LG

Comments ICML 2024

Fewer Tokens and Fewer Videos: Extending Video Understanding Abilities in Large Vision-Language Models

Shimin Chen, Yitian Yuan, Shaoxiang Chen, Zequn Jie, Lin Ma

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.AI

The Neglected Tails in Vision-Language Models

Shubham Parashar, Zhiqiu Lin, Tian Liu, Xiangjue Dong, Yanan Li, Deva Ramanan, James Caverlee, Shu Kong

专题命中 VLM训练与架构 ：vision-language model(title,abstract)；分类 cs.CV、cs.LG

Comments Project Page: https://shubhamprshr27.github.io/neglected-tails-of-vlms/