视觉大模型 / VLM - arXivDaily 专题

2501.08347 2025-01-16 cs.CV cs.AI 62%

SCOT: Self-Supervised Contrastive Pretraining For Zero-Shot Compositional Retrieval

Bhavin Jawade, Joao V. B. Soares, Kapil Thadani, Deen Dayal Mohan, Amir Erfan Eshratifar, Benjamin Culpepper, Paloma de Juan, Srirangaraj Setlur, Venu Govindaraju

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Paper accepted at WACV 2025 in round 1

2501.07978 2025-01-15 cs.CV cs.AI 62%

Facial Dynamics in Video: Instruction Tuning for Improved Facial Expression Perception and Contextual Awareness

Jiaxing Zhao, Boyuan Sun, Xiang Chen, Xihan Wei

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV、cs.AI

2412.18842 2024-12-30 cs.CV cs.LG 62%

Context-Based Semantic-Aware Alignment for Semi-Supervised Multi-Label Learning

Heng-Bo Fan, Ming-Kun Xie, Jia-Hao Xiao, Sheng-Jun Huang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

2412.18150 2024-12-30 cs.CV cs.AI 62%

EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation

Shuhao Han, Haotian Fan, Jiachen Fu, Liang Li, Tao Li, Junhui Cui, Yunqiu Wang, Yang Tai, Jingwei Sun, Chunle Guo, Chongyi Li

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2412.16201 2024-12-24 cs.RO cs.AI cs.LG cs.SY eess.SY 62%

CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning

Erfan Doroudian, Hamid Taghavifar

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.AI、cs.LG

2406.10522 2024-12-19 cs.LG cs.AI cs.CL 62%

Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning

Jifan Zhang, Lalit Jain, Yang Guo, Jiayi Chen, Kuan Lok Zhou, Siddharth Suresh, Andrew Wagenmaker, Scott Sievert, Timothy Rogers, Kevin Jamieson, Robert Mankoff, Robert Nowak

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.AI、cs.LG

2412.10594 2024-12-17 cs.CV cs.LG 62%

Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics

Sara Ghazanfari, Siddharth Garg, Nicolas Flammarion, Prashanth Krishnamurthy, Farshad Khorrami, Francesco Croce

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

2412.10348 2024-12-16 cs.CV cs.AI 62%

A dual contrastive framework

Yuan Sun, Zhao Zhang, Jorge Ortiz

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2412.08979 2024-12-13 cs.LG cs.CV 62%

A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter

Zirun Guo, Xize Cheng, Yangyang Wu, Tao Jin

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted at AAAI 2025

2412.08635 2024-12-12 cs.CL cs.CV cs.LG 62%

Multimodal Latent Language Modeling with Next-Token Diffusion

Yutao Sun, Hangbo Bao, Wenhui Wang, Zhiliang Peng, Li Dong, Shaohan Huang, Jianyong Wang, Furu Wei

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV、cs.LG

2412.05536 2024-12-10 eess.IV cs.AI cs.CL cs.CV 62%

Comprehensive Evaluation of Multimodal AI Models in Medical Imaging Diagnosis: From Data Augmentation to Preference-Based Comparison

Cailian Ruan, Chengyue Huang, Yahe Yang

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI

2407.00978 2024-12-10 cs.AI cs.LG 62%

Hybrid RAG-empowered Multi-modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-based Contract Approach

Cheng Su, Jinbo Wen, Jiawen Kang, Yonghua Wang, Yuanjia Su, Hudan Pan, Zishao Zhong, M. Shamim Hossain

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.AI、cs.LG

Comments 13 pages, 7 figures

2412.04652 2024-12-09 cs.CV cs.AI 62%

Cross-Self KV Cache Pruning for Efficient Vision-Language Inference

Xiaohuan Pei, Tao Huang, Chang Xu

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2412.03467 2024-12-05 cs.CV cs.AI 62%

Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning

Neale Ratzlaff, Man Luo, Xin Su, Vasudev Lal, Phillip Howard

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI

2412.01289 2024-12-05 cs.CV cs.AI 62%

Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion

Zhuokun Chen, Jinwu Hu, Zeshuai Deng, Yufeng Wang, Bohan Zhuang, Mingkui Tan

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV、cs.AI

2409.14607 2024-12-02 cs.CV cs.LG 62%

Patch Ranking: Efficient CLIP by Learning to Rank Local Patches

Cheng-En Wu, Jinhong Lin, Yu Hen Hu, Pedro Morgado

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted by WACV 2025

2411.17891 2024-11-28 cs.CL cs.AI cs.CV 62%

HOPPR Medical-Grade Platform for Medical Imaging AI

Kalina P. Slavkova, Melanie Traughber, Oliver Chen, Robert Bakos, Shayna Goldstein, Dan Harms, Bradley J. Erickson, Khan M. Siddiqui

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.CV、cs.AI

Comments 6 pages, 3 figures

2405.14129 2024-11-26 cs.CL cs.AI cs.CV 62%

AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability

Fei Zhao, Taotian Pang, Chunhui Li, Zhen Wu, Junjie Guo, Shangyu Xing, Xinyu Dai

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV、cs.AI

2411.15099 2024-11-25 cs.CV cs.CL cs.LG 62%

Context-Aware Multimodal Pretraining

Karsten Roth, Zeynep Akata, Dima Damen, Ivana Balažević, Olivier J. Hénaff

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

2411.12713 2024-11-20 cs.CV cs.AI 62%

CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs

Zhehan Kan, Ce Zhang, Zihan Liao, Yapeng Tian, Wenming Yang, Junyuan Xiao, Xu Li, Dongmei Jiang, Yaowei Wang, Qingmin Liao

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2411.06142 2024-11-12 cs.CV cs.AI 62%

Aquila-plus: Prompt-Driven Visual-Language Models for Pixel-Level Remote Sensing Image Understanding

Kaixuan Lu

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.CV、cs.AI

2411.06074 2024-11-12 cs.CV cs.AI 62%

Aquila: A Hierarchically Aligned Visual-Language Model for Enhanced Remote Sensing Image Comprehension

Kaixuan Lu, Ruiqian Zhang, Xiao Huang, Yuxing Xie

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.CV、cs.AI

2405.17871 2024-11-06 cs.CV cs.AI cs.CL 62%

Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment

Xin Xiao, Bohong Wu, Jiacong Wang, Chunyuan Li, Xun Zhou, Haoyuan Guo

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.CV、cs.AI

Comments NeurlPS 2024, Camera ready

2405.10301 2024-11-06 stat.ML cs.AI cs.LG 62%

Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees

Yu Gui, Ying Jin, Zhimei Ren

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.AI、cs.LG

2411.00871 2024-11-05 cs.LG cs.AI q-bio.MN 62%

LLaMo: Large Language Model-based Molecular Graph Assistant

Jinyoung Park, Minseong Bae, Dohwan Ko, Hyunwoo J. Kim

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.AI、cs.LG

Comments NeurIPS 2024

2409.15657 2024-10-31 cs.AI cs.CL cs.LG 62%

M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

Taowen Wang, Yiyang Liu, James Chenhao Liang, junhan zhao, Yiming Cui, Yuning Mao, Shaoliang Nie, Jiahao Liu, Fuli Feng, Zenglin Xu, Cheng Han, Lifu Huang, Qifan Wang, Dongfang Liu

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments EMNLP 2024

2410.19079 2024-10-30 cs.CV cs.LG 62%

BIFRÖST: 3D-Aware Image compositing with Language Instructions

Lingxiao Li, Kaixiong Gong, Weihong Li, Xili Dai, Tao Chen, Xiaojun Yuan, Xiangyu Yue

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV、cs.LG

Comments NeurIPS 2024, Code Available: https://github.com/lingxiao-li/Bifrost

2410.20165 2024-10-29 cs.CV cs.AI 62%

Diff-CXR: Report-to-CXR generation through a disease-knowledge enhanced diffusion model

Peng Huang, Bowen Guo, Shuyu Liang, Junhu Fu, Yuanyuan Wang, Yi Guo

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2407.12315 2024-10-29 cs.CV cs.AI cs.HC cs.IR 62%

ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map

Yilin Ye, Shishi Xiao, Xingchen Zeng, Wei Zeng

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments VIS 2024

2410.15780 2024-10-22 cs.CV cs.AI 62%

An Efficient System for Automatic Map Storytelling -- A Case Study on Historical Maps

Ziyi Liu, Claudio Affolter, Sidi Wu, Yizi Chen, Lorenz Hurni

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

1. VLM训练与架构 5058 篇

SCOT: Self-Supervised Contrastive Pretraining For Zero-Shot Compositional Retrieval

Facial Dynamics in Video: Instruction Tuning for Improved Facial Expression Perception and Contextual Awareness

Context-Based Semantic-Aware Alignment for Semi-Supervised Multi-Label Learning

EvalMuse-40K: A Reliable and Fine-Grained Benchmark with Comprehensive Human Annotations for Text-to-Image Generation Model Evaluation

CLIP-RLDrive: Human-Aligned Autonomous Driving via CLIP-Based Reward Shaping in Reinforcement Learning

Humor in AI: Massive Scale Crowd-Sourced Preferences and Benchmarks for Cartoon Captioning

Towards Unified Benchmark and Models for Multi-Modal Perceptual Metrics

A dual contrastive framework

A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter

Multimodal Latent Language Modeling with Next-Token Diffusion

Comprehensive Evaluation of Multimodal AI Models in Medical Imaging Diagnosis: From Data Augmentation to Preference-Based Comparison

Hybrid RAG-empowered Multi-modal LLM for Secure Data Management in Internet of Medical Things: A Diffusion-based Contract Approach

Cross-Self KV Cache Pruning for Efficient Vision-Language Inference

Training-Free Mitigation of Language Reasoning Degradation After Multimodal Instruction Tuning

Enhancing Perception Capabilities of Multimodal LLMs with Training-Free Fusion

Patch Ranking: Efficient CLIP by Learning to Rank Local Patches

HOPPR Medical-Grade Platform for Medical Imaging AI

AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability

Context-Aware Multimodal Pretraining

CATCH: Complementary Adaptive Token-level Contrastive Decoding to Mitigate Hallucinations in LVLMs

Aquila-plus: Prompt-Driven Visual-Language Models for Pixel-Level Remote Sensing Image Understanding

Aquila: A Hierarchically Aligned Visual-Language Model for Enhanced Remote Sensing Image Comprehension

Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignment

Conformal Alignment: Knowing When to Trust Foundation Models with Guarantees

LLaMo: Large Language Model-based Molecular Graph Assistant

M$^2$PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning

BIFRÖST: 3D-Aware Image compositing with Language Instructions

Diff-CXR: Report-to-CXR generation through a disease-knowledge enhanced diffusion model

ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Map

An Efficient System for Automatic Map Storytelling -- A Case Study on Historical Maps