视觉大模型 / VLM - arXivDaily 专题

2405.01825 2024-08-27 cs.CV 70%

Improving Concept Alignment in Vision-Language Concept Bottleneck Models

Nithish Muthuchamy Selvaraj, Xiaobao Guo, Adams Wai-Kin Kong, Alex Kot

专题命中 VLM训练与架构：vision language model(abstract)；VLM(abstract)；分类 cs.CV

2408.11424 2024-08-22 cs.CV 70%

EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning

Bohao Xing, Zitong Yu, Xin Liu, Kaishen Yuan, Qilang Ye, Weicheng Xie, Huanjing Yue, Jingyu Yang, Heikki Kälviäinen

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

2407.16552 2024-07-25 cs.CV cs.MM 70%

MicroEmo: Time-Sensitive Multimodal Emotion Recognition with Micro-Expression Dynamics in Video Dialogues

Liyun Zhang

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

2407.14885 2024-07-23 cs.CL cs.CV 70%

Falcon2-11B Technical Report

Quentin Malartic, Nilabhra Roy Chowdhury, Ruxandra Cojocaru, Mugariya Farooq, Giulia Campesan, Yasser Abdelaziz Dahou Djilali, Sanath Narayan, Ankit Singh, Maksim Velikanov, Basma El Amel Boussaha, Mohammed Al-Yafeai, Hamza Alobeidli, Leen Al Qadi, Mohamed El Amine Seddik, Kirill Fedyanin, Reda Alami, Hakim Hacid

专题命中 VLM训练与架构：vision language model(abstract)；VLM(abstract)；分类 cs.CV

2407.12025 2024-07-18 cs.HC cs.AI 70%

LLM4DESIGN: An Automated Multi-Modal System for Architectural and Environmental Design

Ran Chen, Xueqi Yao, Xuhui Jiang

专题命中 VLM训练与架构：VLM(abstract)；visual language model(abstract)；分类 cs.AI

2403.12736 2024-07-18 cs.CV 70%

Towards Multimodal In-Context Learning for Vision & Language Models

Sivan Doveh, Shaked Perek, M. Jehanzeb Mirza, Wei Lin, Amit Alfassy, Assaf Arbelle, Shimon Ullman, Leonid Karlinsky

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2407.02218 2024-07-08 cs.CV 70%

Multi-Modal Video Dialog State Tracking in the Wild

Adnen Abdessaied, Lei Shi, Andreas Bulling

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments ECCV 2024

2312.12379 2024-07-08 cs.CV 70%

Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning

Yunhao Gou, Zhili Liu, Kai Chen, Lanqing Hong, Hang Xu, Aoxue Li, Dit-Yan Yeung, James T. Kwok, Yu Zhang

专题命中 VLM训练与架构：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV

Comments Project website: https://gyhdog99.github.io/projects/mocle/

2406.13875 2024-06-26 cs.CV 70%

WATT: Weight Average Test-Time Adaptation of CLIP

David Osowiechi, Mehrdad Noori, Gustavo Adolfo Vargas Hakim, Moslem Yazdanpanah, Ali Bahri, Milad Cheraghalikhani, Sahar Dastani, Farzad Beizaee, Ismail Ben Ayed, Christian Desrosiers

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2406.13787 2024-06-21 cs.RO cs.CV 70%

LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application

Zhe Huang, John Pohovey, Ananya Yammanuru, Katherine Driggs-Campbell

专题命中 VLM训练与架构：vision language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Spotlight Presentation at the 3rd Workshop on Computer Vision in the Wild at CVPR 2024. Also accepted by the 5th Annual Embodied AI Workshop at CVPR 2024

2307.06930 2024-06-21 cs.CV cs.CL 70%

mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs

Gregor Geigle, Abhay Jain, Radu Timofte, Goran Glavaš

专题命中 VLM训练与架构：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV

Comments ALVR Workshop 2024

2406.11896 2024-06-20 cs.LG 70%

DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

Hao Bai, Yifei Zhou, Mert Cemri, Jiayi Pan, Alane Suhr, Sergey Levine, Aviral Kumar

专题命中 VLM训练与架构：vision language model(abstract)；VLM(abstract)；分类 cs.LG

Comments 11 pages of main text, 28 pages in total

2406.08478 2024-06-19 cs.CV cs.CL 70%

What If We Recaption Billions of Web Images with LLaMA-3?

Xianhang Li, Haoqin Tu, Mude Hui, Zeyu Wang, Bingchen Zhao, Junfei Xiao, Sucheng Ren, Jieru Mei, Qing Liu, Huangjie Zheng, Yuyin Zhou, Cihang Xie

专题命中 VLM训练与架构：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV

Comments First five authors contributed equally

2406.07163 2024-06-12 cs.CV 70%

FaceGPT: Self-supervised Learning to Chat about 3D Human Faces

Haoran Wang, Mohit Mendiratta, Christian Theobalt, Adam Kortylewski

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2402.10896 2024-06-04 cs.CV 70%

PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter

Junfei Xiao, Zheng Xu, Alan Yuille, Shen Yan, Boyu Wang

专题命中 VLM训练与架构：vision-language model(abstract)；visual question answering(abstract)；分类 cs.CV

Comments Technical report, 15 pages; v2 fix typos, add additional results in appendix

2405.18361 2024-05-29 cs.CV 70%

Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?

Yifan Bai, Dongming Wu, Yingfei Liu, Fan Jia, Weixin Mao, Ziheng Zhang, Yucheng Zhao, Jianbing Shen, Xing Wei, Tiancai Wang, Xiangyu Zhang

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2402.12741 2024-05-27 cs.CV 70%

MuLan: Multimodal-LLM Agent for Progressive and Interactive Multi-Object Diffusion

Sen Li, Ruochen Wang, Cho-Jui Hsieh, Minhao Cheng, Tianyi Zhou

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Added the application to human-agent interaction; added discussion with concurrent work

2405.04403 2024-05-08 cs.CV cs.CL 70%

Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks

Georgios Pantazopoulos, Amit Parekh, Malvina Nikandrou, Alessandro Suglia

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2404.16385 2024-04-26 cs.CV 70%

Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models

Jiawei Chen, Dingkang Yang, Yue Jiang, Mingcheng Li, Jinjie Wei, Xiaolu Hou, Lihua Zhang

专题命中 VLM训练与架构：visual language model(abstract)；visual question answering(abstract)；分类 cs.CV

2404.09987 2024-04-26 cs.CV 70%

OneChart: Purify the Chart Structural Extraction via One Auxiliary Token

Jinyue Chen, Lingyu Kong, Haoran Wei, Chenglong Liu, Zheng Ge, Liang Zhao, Jianjian Sun, Chunrui Han, Xiangyu Zhang

专题命中 VLM训练与架构：vision-language model(abstract)；LLaVA(abstract)；分类 cs.CV

Comments 14 pages, 9 figures and 6 tables

2404.05674 2024-04-09 cs.CV 70%

MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation

Kunpeng Song, Yizhe Zhu, Bingchen Liu, Qing Yan, Ahmed Elgammal, Xiao Yang

专题命中 VLM训练与架构：multimodal large language model(abstract)；MLLM(abstract)；分类 cs.CV

2404.05621 2024-04-09 cs.CV 70%

MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning

Matteo Farina, Massimiliano Mancini, Elia Cunegatti, Gaowen Liu, Giovanni Iacca, Elisa Ricci

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments CVPR 2024

2404.00851 2024-04-02 cs.CV 70%

Prompt Learning via Meta-Regularization

Jinyoung Park, Juyeon Ko, Hyunwoo J. Kim

专题命中 VLM训练与架构：vision-language model(abstract)；vision language model(abstract)；分类 cs.CV

Comments CVPR 2024

2312.00081 2024-04-02 cs.CV 70%

Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding

Wujian Peng, Sicheng Xie, Zuyao You, Shiyi Lan, Zuxuan Wu

专题命中 VLM训练与架构：vision language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Accepted by CVPR 2024

2311.16714 2024-04-01 cs.CV 70%

Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

Yijun Yang, Tianyi Zhou, Kanxue Li, Dapeng Tao, Lusong Li, Li Shen, Xiaodong He, Jing Jiang, Yuhui Shi

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments Accepted by CVPR 2024

2403.11835 2024-03-19 cs.CV 70%

Agent3D-Zero: An Agent for Zero-shot 3D Understanding

Sha Zhang, Di Huang, Jiajun Deng, Shixiang Tang, Wanli Ouyang, Tong He, Yanyong Zhang

专题命中 VLM训练与架构：VLM(abstract)；visual language model(abstract)；分类 cs.CV

Comments project page: https://zhangsha1024.github.io/Agent3D-Zero/

2403.01209 2024-03-05 cs.CV 70%

Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning

Shuo Yang, Zirui Shang, Yongqi Wang, Derong Deng, Hongwei Chen, Qiyuan Cheng, Xinxiao Wu

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2402.13232 2024-02-21 cs.CV cs.RO 70%

A Touch, Vision, and Language Dataset for Multimodal Alignment

Letian Fu, Gaurav Datta, Huang Huang, William Chung-Ho Panitch, Jaimyn Drake, Joseph Ortiz, Mustafa Mukadam, Mike Lambeta, Roberto Calandra, Ken Goldberg

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

2402.11690 2024-02-20 cs.CL cs.CV 70%

Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning

Zhiyang Xu, Chao Feng, Rulin Shao, Trevor Ashby, Ying Shen, Di Jin, Yu Cheng, Qifan Wang, Lifu Huang

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.CV

Comments 8 Pages, visual instruction tuning

2402.04764 2024-02-08 cs.LG 70%

Code as Reward: Empowering Reinforcement Learning with VLMs

David Venuto, Sami Nur Islam, Martin Klissarov, Doina Precup, Sherry Yang, Ankit Anand

专题命中 VLM训练与架构：vision-language model(abstract)；VLM(abstract)；分类 cs.LG

1. VLM训练与架构 5048 篇

Improving Concept Alignment in Vision-Language Concept Bottleneck Models

EMO-LLaMA: Enhancing Facial Emotion Understanding with Instruction Tuning

MicroEmo: Time-Sensitive Multimodal Emotion Recognition with Micro-Expression Dynamics in Video Dialogues

Falcon2-11B Technical Report

LLM4DESIGN: An Automated Multi-Modal System for Architectural and Environmental Design

Towards Multimodal In-Context Learning for Vision & Language Models

Multi-Modal Video Dialog State Tracking in the Wild

Mixture of Cluster-conditional LoRA Experts for Vision-language Instruction Tuning

WATT: Weight Average Test-Time Adaptation of CLIP

LIT: Large Language Model Driven Intention Tracking for Proactive Human-Robot Collaboration -- A Robot Sous-Chef Application

mBLIP: Efficient Bootstrapping of Multilingual Vision-LLMs

DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learning

What If We Recaption Billions of Web Images with LLaMA-3?

FaceGPT: Self-supervised Learning to Chat about 3D Human Faces

PaLM2-VAdapter: Progressively Aligned Language Model Makes a Strong Vision-language Adapter

Is a 3D-Tokenized LLM the Key to Reliable Autonomous Driving?

MuLan: Multimodal-LLM Agent for Progressive and Interactive Multi-Object Diffusion

Learning To See But Forgetting To Follow: Visual Instruction Tuning Makes LLMs More Prone To Jailbreak Attacks

Efficiency in Focus: LayerNorm as a Catalyst for Fine-tuning Medical Visual Language Pre-trained Models

OneChart: Purify the Chart Structural Extraction via One Auxiliary Token

MoMA: Multimodal LLM Adapter for Fast Personalized Image Generation

MULTIFLOW: Shifting Towards Task-Agnostic Vision-Language Pruning

Prompt Learning via Meta-Regularization

Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding

Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld

Agent3D-Zero: An Agent for Zero-shot 3D Understanding

Data-free Multi-label Image Recognition via LLM-powered Prompt Tuning

A Touch, Vision, and Language Dataset for Multimodal Alignment

Vision-Flan: Scaling Human-Labeled Tasks in Visual Instruction Tuning

Code as Reward: Empowering Reinforcement Learning with VLMs