视觉大模型 / VLM - arXivDaily 专题

2410.12051 2024-10-17 cs.HC cs.AI cs.ET cs.MM 57%

Enabling Data-Driven and Empathetic Interactions: A Context-Aware 3D Virtual Agent in Mixed Reality for Enhanced Financial Customer Experience

Cindy Xu, Mengyu Chen, Pranav Deshpande, Elvir Azanli, Runqing Yang, Joseph Ligman

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.AI

Comments to appear at 1st Workshop on Intelligent XR: Harnessing AI for Next-Generation XR User Experiences at International Symposium on Mixed and Augmented Reality (ISMAR) 2024

2410.11417 2024-10-16 cs.CV cs.MM 57%

VidCompress: Memory-Enhanced Temporal Compression for Video Understanding in Large Language Models

Xiaohan Lan, Yitian Yuan, Zequn Jie, Lin Ma

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV

Comments 9 pages, 4 figures

2410.11255 2024-10-16 cs.CV 57%

CLIP-DFGS: A Hard Sample Mining Method for CLIP in Generalizable Person Re-Identification

Huazhong Zhao, Lei Qi, Xin Geng

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

Comments Accepted by ACM TOMM

2410.10319 2024-10-15 cs.CV cs.MM 57%

Spatial-Aware Efficient Projector for MLLMs via Multi-Layer Feature Aggregation

Shun Qian, Bingquan Liu, Chengjie Sun, Zhen Xu, Baoxun Wang

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV

Comments 10 pages, 3 figures

2410.08791 2024-10-14 cs.LG 57%

Superpipeline: A Universal Approach for Reducing GPU Memory Usage in Large Models

Reza Abbasi, Sernam Lim

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.LG

2402.19085 2024-10-14 cs.CL cs.AI cs.SY eess.SY 57%

Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment

Yiju Guo, Ganqu Cui, Lifan Yuan, Ning Ding, Zexu Sun, Bowen Sun, Huimin Chen, Ruobing Xie, Jie Zhou, Yankai Lin, Zhiyuan Liu, Maosong Sun

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.AI

Comments EMNLP 2024 main conference

2410.07073 2024-10-14 cs.CV cs.CL 57%

Pixtral 12B

Pravesh Agrawal, Szymon Antoniak, Emma Bou Hanna, Baptiste Bout, Devendra Chaplot, Jessica Chudnovsky, Diogo Costa, Baudouin De Monicault, Saurabh Garg, Theophile Gervet, Soham Ghosh, Amélie Héliou, Paul Jacob, Albert Q. Jiang, Kartik Khandelwal, Timothée Lacroix, Guillaume Lample, Diego Las Casas, Thibaut Lavril, Teven Le Scao, Andy Lo, William Marshall, Louis Martin, Arthur Mensch, Pavankumar Muddireddy, Valera Nemychnikova, Marie Pellat, Patrick Von Platen, Nikhil Raghuraman, Baptiste Rozière, Alexandre Sablayrolles, Lucile Saulnier, Romain Sauvestre, Wendy Shang, Roman Soletskyi, Lawrence Stewart, Pierre Stock, Joachim Studnia, Sandeep Subramanian, Sagar Vaze, Thomas Wang, Sophia Yang

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

2410.07648 2024-10-11 cs.CV 57%

FLIER: Few-shot Language Image Models Embedded with Latent Representations

Zhinuo Zhou, Peng Zhou, Xiaoyong Pan

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

Comments 8 pages,3 figures

2405.16785 2024-10-11 cs.CV 57%

PromptFix: You Prompt and We Fix the Photo

Yongsheng Yu, Ziyun Zeng, Hang Hua, Jianlong Fu, Jiebo Luo

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

Comments Accepted to NeurIPS 2024

2403.09559 2024-10-11 cs.CL cs.CV 57%

Less is More: High-value Data Selection for Visual Instruction Tuning

Zikang Liu, Kun Zhou, Wayne Xin Zhao, Dawei Gao, Yaliang Li, Ji-Rong Wen

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.CV

Comments Under Review

2410.07113 2024-10-10 cs.CV 57%

Personalized Visual Instruction Tuning

Renjie Pi, Jianshu Zhang, Tianyang Han, Jipeng Zhang, Rui Pan, Tong Zhang

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV

2410.04751 2024-10-08 cs.CV cs.CL 57%

Intriguing Properties of Large Language and Vision Models

Young-Jun Lee, Byungsoo Ko, Han-Gyu Kim, Yechan Hwang, Ho-Jin Choi

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV

Comments Code is available in https://github.com/passing2961/IP-LLVM

2403.13248 2024-10-07 cs.CV 57%

Mora: Enabling Generalist Video Generation via A Multi-Agent Framework

Zhengqing Yuan, Yixin Liu, Yihan Cao, Weixiang Sun, Haolong Jia, Ruoxi Chen, Zhaoxu Li, Bin Lin, Li Yuan, Lifang He, Chi Wang, Yanfang Ye, Lichao Sun

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV

2410.02049 2024-10-04 cs.CV cs.CL cs.GR 57%

Emo3D: Metric and Benchmarking Dataset for 3D Facial Expression Generation from Emotion Description

Mahshid Dehghani, Amirahmad Shafiee, Ali Shafiei, Neda Fallah, Farahmand Alizadeh, Mohammad Mehdi Gholinejad, Hamid Behroozi, Jafar Habibi, Ehsaneddin Asgari

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

Comments 11 pages, 10 figures

2406.11262 2024-10-04 cs.CV 57%

Generative Visual Instruction Tuning

Jefferson Hernandez, Ruben Villegas, Vicente Ordonez

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV

Comments Add more results using task tokens, expand the introduction and related work FIX: error in LLM-as-judge evaluation that was over-inflating the results

2410.00905 2024-10-02 cs.CV 57%

Removing Distributional Discrepancies in Captions Improves Image-Text Alignment

Yuheng Li, Haotian Liu, Mu Cai, Yijun Li, Eli Shechtman, Zhe Lin, Yong Jae Lee, Krishna Kumar Singh

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV

2409.19961 2024-10-01 cs.CV cs.CL 57%

Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval

Yabing Wang, Le Wang, Qiang Zhou, Zhibin Wang, Hao Li, Gang Hua, Wei Tang

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV

Comments Accepted by ACM Multimedia

2409.19806 2024-10-01 cs.SD cs.AI eess.AS 57%

PALM: Few-Shot Prompt Learning for Audio Language Models

Asif Hanif, Maha Tufail Agro, Mohammad Areeb Qazi, Hanan Aldarmaki

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.AI

Comments EMNLP 2024 (Main)

2312.08168 2024-10-01 cs.CV 57%

Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers

Haifeng Huang, Yilun Chen, Zehan Wang, Rongjie Huang, Runsen Xu, Tai Wang, Luping Liu, Xize Cheng, Yang Zhao, Jiangmiao Pang, Zhou Zhao

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.CV

2409.18869 2024-09-30 cs.CV 57%

Emu3: Next-Token Prediction is All You Need

Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, Yingli Zhao, Yulong Ao, Xuebin Min, Tao Li, Boya Wu, Bo Zhao, Bowen Zhang, Liangdong Wang, Guang Liu, Zheqi He, Xi Yang, Jingjing Liu, Yonghua Lin, Tiejun Huang, Zhongyuan Wang

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV

Comments Project Page: https://emu.baai.ac.cn

2409.17727 2024-09-27 cs.RO cs.CV 57%

Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applications

Nghia Nguyen, Minh Nhat Vu, Tung D. Ta, Baoru Huang, Thieu Vo, Ngan Le, Anh Nguyen

专题命中 VLM训练与架构：vision language model(abstract)；分类 cs.CV

Comments 7 pages

2409.17106 2024-09-26 cs.CV cs.GR 57%

Text2CAD: Generating Sequential CAD Models from Beginner-to-Expert Level Text Prompts

Mohammad Sadil Khan, Sankalp Sinha, Talha Uddin Sheikh, Didier Stricker, Sk Aziz Ali, Muhammad Zeshan Afzal

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV

Comments Accepted in NeurIPS 2024 (Spotlight)

2409.16145 2024-09-25 cs.CV 57%

Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment

Yuxiao Chen, Kai Li, Wentao Bao, Deep Patel, Yu Kong, Martin Renqiang Min, Dimitris N. Metaxas

专题命中 VLM训练与架构：grounding(abstract)；分类 cs.CV

Comments Accepted to ECCV 2024

2409.16084 2024-09-25 cs.CV 57%

MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarios

Jiacheng Ruan, Wenzhen Yuan, Zehao Lin, Ning Liao, Zhiyu Li, Feiyu Xiong, Ting Liu, Yuzhuo Fu

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV

Comments 9 pages, 5 figures. Work in progress

2409.15292 2024-09-25 cs.RO cs.AI 57%

SketcherX: AI-Driven Interactive Robotic drawing with Diffusion model and Vectorization Techniques

Jookyung Song, Mookyoung Kang, Nojun Kwak

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.AI

Comments 10 pages, 10 figures

2402.19404 2024-09-23 cs.CV cs.CL 57%

EAMA : Entity-Aware Multimodal Alignment Based Approach for News Image Captioning

Junzhe Zhang, Huixuan Zhang, Xunjian Yin, Xiaojun Wan

专题命中 VLM训练与架构：MLLM(abstract)；分类 cs.CV

2409.11182 2024-09-18 cs.CV 57%

Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Driving

Yunsheng Ma, Amr Abdelraouf, Rohit Gupta, Ziran Wang, Kyungtae Han

专题命中 VLM训练与架构：multimodal large language model(abstract)；分类 cs.CV

Comments 10 pages, 3 figures, 4 tables

2308.16463 2024-09-18 cs.CV cs.CL 57%

Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models

Yupan Huang, Zaiqiao Meng, Fangyu Liu, Yixuan Su, Nigel Collier, Yutong Lu

专题命中 VLM训练与架构：LLaVA(abstract)；分类 cs.CV

Comments ICLR 2024 Workshop (Navigating and Addressing Data Problems for Foundation Models)

2304.05653 2024-09-17 cs.CV 57%

A Closer Look at the Explainability of Contrastive Language-Image Pre-training

Yi Li, Hualiang Wang, Yiqun Duan, Jiheng Zhang, Xiaomeng Li

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

Comments 30 pages, 11 figures, under review

2409.08598 2024-09-16 cs.CV 57%

Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation

Hangyu Li, Yihan Xu, Jiangchao Yao, Nannan Wang, Xinbo Gao, Bo Han

专题命中 VLM训练与架构：vision-language model(abstract)；分类 cs.CV

1. VLM训练与架构 5058 篇

Enabling Data-Driven and Empathetic Interactions: A Context-Aware 3D Virtual Agent in Mixed Reality for Enhanced Financial Customer Experience

VidCompress: Memory-Enhanced Temporal Compression for Video Understanding in Large Language Models

CLIP-DFGS: A Hard Sample Mining Method for CLIP in Generalizable Person Re-Identification

Spatial-Aware Efficient Projector for MLLMs via Multi-Layer Feature Aggregation

Superpipeline: A Universal Approach for Reducing GPU Memory Usage in Large Models

Controllable Preference Optimization: Toward Controllable Multi-Objective Alignment

Pixtral 12B

FLIER: Few-shot Language Image Models Embedded with Latent Representations

PromptFix: You Prompt and We Fix the Photo

Less is More: High-value Data Selection for Visual Instruction Tuning

Personalized Visual Instruction Tuning

Intriguing Properties of Large Language and Vision Models

Mora: Enabling Generalist Video Generation via A Multi-Agent Framework

Emo3D: Metric and Benchmarking Dataset for 3D Facial Expression Generation from Emotion Description

Generative Visual Instruction Tuning

Removing Distributional Discrepancies in Captions Improves Image-Text Alignment

Multimodal LLM Enhanced Cross-lingual Cross-modal Retrieval

PALM: Few-Shot Prompt Learning for Audio Language Models

Chat-Scene: Bridging 3D Scene and Large Language Models with Object Identifiers

Emu3: Next-Token Prediction is All You Need

Robotic-CLIP: Fine-tuning CLIP on Action Data for Robotic Applications

Text2CAD: Generating Sequential CAD Models from Beginner-to-Expert Level Text Prompts

Learning to Localize Actions in Instructional Videos with LLM-Based Multi-Pathway Text-Video Alignment

MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarios

SketcherX: AI-Driven Interactive Robotic drawing with Diffusion model and Vectorization Techniques

EAMA : Entity-Aware Multimodal Alignment Based Approach for News Image Captioning

Video Token Sparsification for Efficient Multimodal LLMs in Autonomous Driving

Sparkles: Unlocking Chats Across Multiple Images for Multimodal Instruction-Following Models

A Closer Look at the Explainability of Contrastive Language-Image Pre-training

Knowledge-Enhanced Facial Expression Recognition with Emotional-to-Neutral Transformation