视觉大模型 / VLM

How Well Do Multi-modal LLMs Interpret CT Scans? An Auto-Evaluation Framework for Analyses

Qingqing Zhu, Benjamin Hou, Tejas S. Mathai, Pritam Mukherjee, Qiao Jin, Xiuying Chen, Zhizheng Wang, Ruida Cheng, Ronald M. Summers, Zhiyong Lu

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI

MINT: a Multi-modal Image and Narrative Text Dubbing Dataset for Foley Audio Content Planning and Generation

Ruibo Fu, Shuchen Shi, Hongming Guo, Tao Wang, Chunyu Qiang, Zhengqi Wen, Jianhua Tao, Xin Qi, Yi Lu, Xiaopeng Wang, Zhiyong Wang, Yukun Liu, Xuefei Liu, Shuai Zhang, Guanjun Li

专题命中 VLM训练与架构 ：LLaVA(abstract)；分类 cs.CV、cs.AI

Direct Preference Optimization for Suppressing Hallucinated Prior Exams in Radiology Report Generation

Oishi Banerjee, Hong-Yu Zhou, Subathra Adithan, Stephen Kwak, Kay Wu, Pranav Rajpurkar

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Added acknowledgemnts

SViTT-Ego: A Sparse Video-Text Transformer for Egocentric Video

Hector A. Valdez, Kyle Min, Subarna Tripathi

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

CPLIP: Zero-Shot Learning for Histopathology with Comprehensive Vision-Language Alignment

Sajid Javed, Arif Mahmood, Iyyakutti Iyappan Ganapathi, Fayaz Ali Dharejo, Naoufel Werghi, Mohammed Bennamoun

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Synth$^2$: Boosting Visual-Language Models with Synthetic Captions and Image Embeddings

Sahand Sharifzadeh, Christos Kaplanis, Shreya Pathak, Dharshan Kumaran, Anastasija Ilic, Jovana Mitrovic, Charles Blundell, Andrea Banino

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI

Comments 9 pages, 6 figures

Prompt-based Visual Alignment for Zero-shot Policy Transfer

Haihan Gao, Rui Zhang, Qi Yi, Hantao Yao, Haochen Li, Jiaming Guo, Shaohui Peng, Yunkai Gao, QiCheng Wang, Xing Hu, Yuanbo Wen, Zihao Zhang, Zidong Du, Ling Li, Qi Guo, Yunji Chen

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI

Comments This paper has been accepted by ICML2024

Exploiting LMM-based knowledge for image classification tasks

Maria Tzelepi, Vasileios Mezaris

专题命中 VLM训练与架构 ：vision language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted for publication, 25th Int. Conf. on Engineering Applications of Neural Networks (EANN/EAAAI 2024), Corfu, Greece, June 2024. This is the "submitted manuscript"

Multimodal Deep Learning for Low-Resource Settings: A Vector Embedding Alignment Approach for Healthcare Applications

David Restrepo, Chenwei Wu, Sebastián Andrés Cajas, Luis Filipe Nakayama, Leo Anthony Celi, Diego M López

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.AI、cs.LG

WIDIn: Wording Image for Domain-Invariant Representation in Single-Source Domain Generalization

Jiawei Ma, Yulei Niu, Shiyuan Huang, Guangxing Han, Shih-Fu Chang

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Explaining Multi-modal Large Language Models by Analyzing their Vision Perception

Loris Giulivi, Giacomo Boracchi

专题命中 VLM训练与架构 ：MLLM(abstract)；分类 cs.CV、cs.AI

Comments Submitted at BMVC 2024

LDEdit: Towards Generalized Text Guided Image Manipulation via Latent Diffusion Models

Paramanand Chandramouli, Kanchana Vaishnavi Gandikota

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted BMVC 2022

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, Yinfei Yang

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.LG

CFIR: Fast and Effective Long-Text To Image Retrieval for Large Corpora

Zijun Long, Xuri Ge, Richard Mccreadie, Joemon Jose

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.AI

TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification

Qinying Liu, Wei Wu, Kecheng Zheng, Zhan Tong, Jiawei Liu, Yu Liu, Wei Chen, Zilei Wang, Yujun Shen

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

VidLA: Video-Language Alignment at Scale

Mamshad Nayeem Rizve, Fan Fei, Jayakrishnan Unnikrishnan, Son Tran, Benjamin Z. Yao, Belinda Zeng, Mubarak Shah, Trishul Chilimbi

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.CV、cs.LG

Comments Accepted to CVPR 2024

Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuning

Cristina Menghini, Andrew Delworth, Stephen H. Bach

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

KG-TREAT: Pre-training for Treatment Effect Estimation by Synergizing Patient Data with Knowledge Graphs

Ruoqi Liu, Lingfei Wu, Ping Zhang

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.AI、cs.LG

Comments AAAI 2024 Main Track

Referee Can Play: An Alternative Approach to Conditional Generation via Model Inversion

Xuantong Liu, Tianyang Hu, Wenjia Wang, Kenji Kawaguchi, Yuan Yao

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.AI、cs.LG

LLMs as Visual Explainers: Advancing Image Classification with Evolving Visual Descriptions

Songhao Han, Le Zhuo, Yue Liao, Si Liu

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

Emergent Communication Pretraining for Few-Shot Machine Translation

Yaoyiran Li, Edoardo M. Ponti, Ivan Vulić, Anna Korhonen

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.AI、cs.LG

Journal ref Proceedings of the 28th International Conference on Computational Linguistics (2020) pages 4716-4731

M2-Encoder: Advancing Bilingual Image-Text Understanding by Large-scale Efficient Pretraining

Qingpei Guo, Furong Xu, Hanxiao Zhang, Wang Ren, Ziping Ma, Lin Ju, Jian Wang, Jingdong Chen, Ming Yang

专题命中 VLM训练与架构 ：VLM(abstract)；分类 cs.CV、cs.AI

Vision-Language Foundation Models as Effective Robot Imitators

Xinghang Li, Minghuan Liu, Hanbo Zhang, Cunjun Yu, Jie Xu, Hongtao Wu, Chilam Cheang, Ya Jing, Weinan Zhang, Huaping Liu, Hang Li, Tao Kong

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.AI、cs.LG

Comments Fix typos. Project page: https://roboflamingo.github.io

Zoom-shot: Fast and Efficient Unsupervised Zero-Shot Transfer of CLIP to Vision Encoders with Multimodal Loss

Jordan Shipard, Arnold Wiliem, Kien Nguyen Thanh, Wei Xiang, Clinton Fookes

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments 15 pages

An Autoregressive Text-to-Graph Framework for Joint Entity and Relation Extraction

Urchade Zaratiana, Nadi Tomeh, Pierre Holat, Thierry Charnois

专题命中 VLM训练与架构 ：grounding(abstract)；分类 cs.AI、cs.LG

Comments AAAI 2024 (camera ready version)

Prompt Decoupling for Text-to-Image Person Re-identification

Weihao Li, Lei Tan, Pingyang Dai, Yan Zhang

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

StableLLaVA: Enhanced Visual Instruction Tuning with Synthesized Image-Dialogue Data

Yanda Li, Chi Zhang, Gang Yu, Zhibin Wang, Bin Fu, Guosheng Lin, Chunhua Shen, Ling Chen, Yunchao Wei

专题命中 VLM训练与架构 ：multimodal large language model(abstract)；分类 cs.CV、cs.LG

Comments Project page: https://github.com/icoz69/StableLLAVA

Consensus, dissensus and synergy between clinicians and specialist foundation models in radiology report generation

Ryutaro Tanno, David G. T. Barrett, Andrew Sellergren, Sumedh Ghaisas, Sumanth Dathathri, Abigail See, Johannes Welbl, Karan Singhal, Shekoofeh Azizi, Tao Tu, Mike Schaekermann, Rhys May, Roy Lee, SiWai Man, Zahra Ahmed, Sara Mahdavi, Yossi Matias, Joelle Barral, Ali Eslami, Danielle Belgrave, Vivek Natarajan, Shravya Shetty, Pushmeet Kohli, Po-Sen Huang, Alan Karthikesalingam, Ira Ktena

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.LG

UniDCP: Unifying Multiple Medical Vision-language Tasks via Dynamic Cross-modal Learnable Prompts

Chenlu Zhan, Yufei Zhang, Yu Lin, Gaoang Wang, Hongwei Wang

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Representing visual classification as a linear combination of words

Shobhit Agarwal, Yevgeniy R. Semenov, William Lotter

专题命中 VLM训练与架构 ：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments To be published in the Proceedings of the 3rd Machine Learning for Health symposium, Proceedings of Machine Learning Research (PMLR)