视觉大模型 / VLM - arXivDaily 专题

2406.01662 2024-10-18 cs.CV cs.AI 62%

Few-Shot Classification of Interactive Activities of Daily Living (InteractADL)

Zane Durante, Robathan Harries, Edward Vendrow, Zelun Luo, Yuta Kyuragi, Kazuki Kozuka, Li Fei-Fei, Ehsan Adeli

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV、cs.AI

2410.08469 2024-10-17 cs.LG cs.CL cs.CV 62%

Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP

Eunji Kim, Kyuhong Shim, Simyung Chang, Sungroh Yoon

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted at EMNLP 2024 Findings

2410.05997 2024-10-10 eess.AS cs.CV cs.LG cs.SD 62%

An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment

Hugo Malard, Michel Olvera, Stéphane Lathuiliere, Slim Essid

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV、cs.LG

2410.05938 2024-10-10 cs.CV cs.AI 62%

EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment

Yifei Xing, Xiangyuan Lan, Ruiping Wang, Dongmei Jiang, Wenjun Huang, Qingfang Zheng, Yaowei Wang

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV、cs.AI

2407.12871 2024-10-10 cs.CL cs.AI cs.LG 62%

MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation

Xiaohan Wang, Dian Li, Yilin Zhao, Sinbadliu, Hui Wang

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.AI、cs.LG

Comments 9 pages, 4 figures

2410.05361 2024-10-10 cs.LG cs.AI cs.SD eess.AS 62%

RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction

Yuwei Zhang, Tong Xia, Aaqib Saeed, Cecilia Mascolo

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.AI、cs.LG

2410.03038 2024-10-08 cs.LG cs.CV 62%

CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification

Jinghao Shi, Xiang Shen, Kaili Zhao, Xuedong Wang, Vera Wen, Zixuan Wang, Yifan Wu, Zhixin Zhang

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV、cs.LG

Comments Camera ready for CIKM 2024

2410.03334 2024-10-07 cs.CV cs.AI 62%

An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation

Ahmed Abdulaal, Hugo Fry, Nina Montaña-Brown, Ayodeji Ijishakin, Jack Gao, Stephanie Hyland, Daniel C. Alexander, Daniel C. Castro

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2407.08044 2024-09-30 cs.CL cs.AI cs.LG 62%

RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization

Xijie Huang, Zechun Liu, Shih-Yang Liu, Kwang-Ting Cheng

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.AI、cs.LG

Comments EMNLP 2024 Findings, Codes: https://github.com/HuangOwen/RoLoRA, Models: https://huggingface.co/collections/ScarletAce/rolora-66f5f228a90681c7c4512b28

2409.17634 2024-09-27 cs.CV cs.AI 62%

P4Q: Learning to Prompt for Quantization in Visual-language Models

Huixin Sun, Runqi Wang, Yanjing Li, Xianbin Cao, Xiaolong Jiang, Yao Hu, Baochang Zhang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2407.00634 2024-09-25 cs.CV cs.LG 62%

Tarsier: Recipes for Training and Evaluating Large Video Description Models

Jiawei Wang, Liping Yuan, Yuchen Zhang, Haomiao Sun

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.CV、cs.LG

2409.14163 2024-09-24 cs.CV cs.CL cs.LG 62%

PromptTA: Prompt-driven Text Adapter for Source-free Domain Generalization

Haoran Zhang, Shuanghao Bai, Wanqi Zhou, Jingwen Fu, Badong Chen

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

2409.11059 2024-09-19 cs.CV cs.LG 62%

OneEncoder: A Lightweight Framework for Progressive Alignment of Modalities

Bilal Faye, Hanane Azzag, Mustapha Lebbah

专题命中 VLM训练与架构：visual question answering(abstract)；分类 cs.CV、cs.LG

2409.10921 2024-09-18 cs.CV cs.AI 62%

KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph

Yanbei Jiang, Krista A. Ehinger, Jey Han Lau

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted at IJCAI 2024

2406.18790 2024-09-13 cs.CV cs.AI 62%

MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data

William Berman, Alexander Peysakhovich

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2312.10692 2024-09-04 cs.CV cs.AI 62%

Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion

Xiao Wang, Jiandong Jin, Chenglong Li, Jin Tang, Cheng Zhang, Wei Wang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted by IEEE TCSVT 2024, Camera Ready Version

2408.16667 2024-08-30 cs.LG cs.AI cs.CL cs.MA 62%

Iterative Graph Alignment

Fangyuan Yu, Hardeep Singh Arora, Matt Johnson

专题命中 VLM训练与架构：VLM(abstract)；分类 cs.AI、cs.LG

Comments 12 pages, 4 figures

2408.14023 2024-08-27 cs.CV cs.AI 62%

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

Jiajun Fei, Dian Li, Zhidong Deng, Zekun Wang, Gang Liu, Hui Wang

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 5 figures

2408.07037 2024-08-14 cs.CV cs.AI 62%

PathInsight: Instruction Tuning of Multimodal Datasets and Models for Intelligence Assisted Diagnosis in Histopathology

Xiaomin Wu, Rui Xu, Pengchen Wei, Wenkang Qin, Peixiang Huang, Ziheng Li, Lin Luo

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 2 figures

2408.03574 2024-08-08 cs.CV cs.CL cs.LG 62%

Teach CLIP to Develop a Number Sense for Ordinal Regression

Yao Du, Qiang Zhai, Weihang Dai, Xiaomeng Li

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted by ECCV 2024

2408.00290 2024-08-02 cs.CV cs.AI 62%

Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network

Bin Cheng, Jiaxuan Lu

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV、cs.AI

2403.19584 2024-07-31 cs.CV cs.AI 62%

Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented Generation

Zhongliang Zhou, Jielu Zhang, Zihan Guan, Mengxuan Hu, Ni Lao, Lan Mu, Sheng Li, Gengchen Mai

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV、cs.AI

2406.04413 2024-07-25 cs.CV cs.AI 62%

Efficient 3D-Aware Facial Image Editing via Attribute-Specific Prompt Learning

Amandeep Kumar, Muhammad Awais, Sanath Narayan, Hisham Cholakkal, Salman Khan, Rao Muhammad Anwer

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

Comments Accepted at ECCV, 2024. Amandeep Kumar and Muhammad Awais are joint first authors. More details are available at https://awaisrauf.github.io/3d_face_editing

2310.08320 2024-07-24 cs.LG cs.CL cs.CR cs.CV 62%

Defending Our Privacy With Backdoors

Dominik Hintersdorf, Lukas Struppek, Daniel Neider, Kristian Kersting

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments Accepted at ECAI 2024

2308.12919 2024-07-19 cs.CV cs.LG 62%

Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization

Jian Liang, Lijun Sheng, Zhengbo Wang, Ran He, Tieniu Tan

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments ICML 2024 Highlight

2407.07360 2024-07-11 cs.CV cs.LG 62%

Towards a text-based quantitative and explainable histopathology image analysis

Anh Tien Nguyen, Trinh Thi Le Vuong, Jin Tae Kwak

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.LG

Comments MICCAI 2024 - Early acceptance (Top 11%)

2407.06189 2024-07-09 cs.CV cs.AI 62%

Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision

Orr Zohar, Xiaohan Wang, Yonatan Bitton, Idan Szpektor, Serena Yeung-Levy

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.CV、cs.AI

Comments Project page: https://orrzohar.github.io/projects/video-star/

2407.01791 2024-07-03 cs.CV cs.AI 62%

μ-Bench: A Vision-Language Benchmark for Microscopy Understanding

Alejandro Lozano, Jeffrey Nirschl, James Burgess, Sanket Rajan Gupte, Yuhui Zhang, Alyssa Unell, Serena Yeung-Levy

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV、cs.AI

2309.05519 2024-06-26 cs.AI cs.CL cs.LG 62%

NExT-GPT: Any-to-Any Multimodal LLM

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, Tat-Seng Chua

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.AI、cs.LG

Comments ICML 2024 (Oral)

2406.13193 2024-06-21 cs.LG cs.AI cs.CL physics.chem-ph 62%

PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomes

He Cao, Yanjun Shao, Zhiyuan Liu, Zijing Liu, Xiangru Tang, Yuan Yao, Yu Li

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.AI、cs.LG

1. VLM训练与架构 5058 篇

Few-Shot Classification of Interactive Activities of Daily Living (InteractADL)

Semantic Token Reweighting for Interpretable and Controllable Text Embeddings in CLIP

An Eye for an Ear: Zero-shot Audio Description Leveraging an Image Captioner using Audiovisual Distribution Alignment

EMMA: Empowering Multi-modal Mamba with Structural and Hierarchical Alignment

MetaTool: Facilitating Large Language Models to Master Tools with Meta-task Augmentation

RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction

CPFD: Confidence-aware Privileged Feature Distillation for Short Video Classification

An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generation

RoLoRA: Fine-tuning Rotated Outlier-free LLMs for Effective Weight-Activation Quantization

P4Q: Learning to Prompt for Quantization in Visual-language Models

Tarsier: Recipes for Training and Evaluating Large Video Description Models

PromptTA: Prompt-driven Text Adapter for Source-free Domain Generalization

OneEncoder: A Lightweight Framework for Progressive Alignment of Modalities

KALE: An Artwork Image Captioning System Augmented with Heterogeneous Graph

MUMU: Bootstrapping Multimodal Image Generation from Text-to-Image Data

Pedestrian Attribute Recognition via CLIP based Prompt Vision-Language Fusion

Iterative Graph Alignment

Video-CCAM: Enhancing Video-Language Understanding with Causal Cross-Attention Masks for Short and Long Videos

PathInsight: Instruction Tuning of Multimodal Datasets and Models for Intelligence Assisted Diagnosis in Histopathology

Teach CLIP to Develop a Number Sense for Ordinal Regression

Multi-Modal Parameter-Efficient Fine-tuning via Graph Neural Network

Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented Generation

Efficient 3D-Aware Facial Image Editing via Attribute-Specific Prompt Learning

Defending Our Privacy With Backdoors

Realistic Unsupervised CLIP Fine-tuning with Universal Entropy Optimization

Towards a text-based quantitative and explainable histopathology image analysis

Video-STaR: Self-Training Enables Video Instruction Tuning with Any Supervision

μ-Bench: A Vision-Language Benchmark for Microscopy Understanding

NExT-GPT: Any-to-Any Multimodal LLM

PRESTO: Progressive Pretraining Enhances Synthetic Chemistry Outcomes