多模态大模型 - arXivDaily 专题

2403.17995 2024-03-28 cs.CV cs.AI cs.LG 73%

Semi-Supervised Image Captioning Considering Wasserstein Graph Matching

Yang Yang

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

2312.05588 2024-03-05 cs.AI cs.CV 73%

Language-assisted Vision Model Debugger: A Sample-Free Approach to Finding and Fixing Bugs

Chaoquan Jiang, Jinqiang Wang, Rui Hu, Jitao Sang

专题命中图文多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments 10 pages,8 figures

2312.08846 2024-02-27 cs.LG cs.CL cs.CV 73%

TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training

Chaoya Jiang, Wei ye, Haiyang Xu, Qinghao Ye, Ming Yan, Ji Zhang, Shikun Zhang

专题命中图文多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted on AAAI2024

2402.13936 2024-02-22 cs.CL cs.CV 73%

Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning

Antoine Chaffin, Ewa Kijak, Vincent Claveau

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

2305.15072 2024-02-20 cs.CV cs.MM 73%

PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology

Yuxuan Sun, Chenglu Zhu, Sunyi Zheng, Kai Zhang, Lin Sun, Zhongyi Shui, Yunlong Zhang, Honglin Li, Lin Yang

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Comments 9 pages, 7 figures

2401.16420 2024-01-30 cs.CV cs.CL 73%

InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

Xiaoyi Dong, Pan Zhang, Yuhang Zang, Yuhang Cao, Bin Wang, Linke Ouyang, Xilin Wei, Songyang Zhang, Haodong Duan, Maosong Cao, Wenwei Zhang, Yining Li, Hang Yan, Yang Gao, Xinyue Zhang, Wei Li, Jingwen Li, Kai Chen, Conghui He, Xingcheng Zhang, Yu Qiao, Dahua Lin, Jiaqi Wang

专题命中图文多模态：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments Code and models are available at https://github.com/InternLM/InternLM-XComposer

2311.15826 2023-11-28 cs.CV cs.AI 73%

GeoChat: Grounded Large Vision-Language Model for Remote Sensing

Kartik Kuckreja, Muhammad Sohail Danish, Muzammal Naseer, Abhijit Das, Salman Khan, Fahad Shahbaz Khan

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments 10 pages, 4 figures

2210.04135 2023-10-31 cs.CV cs.LG cs.MM 73%

VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment

Shraman Pramanick, Li Jing, Sayan Nag, Jiachen Zhu, Hardik Shah, Yann LeCun, Rama Chellappa

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Comments Published in TMLR 2023

2303.11403 2023-10-30 cs.CV cs.CL cs.LG 73%

eP-ALM: Efficient Perceptual Augmentation of Language Models

Mustafa Shukor, Corentin Dancette, Matthieu Cord

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted at ICCV 2023. Project page: https://mshukor.github.io/eP-ALM.github.io/

2310.04550 2023-10-10 cs.CV cs.CL cs.LG 73%

Module-wise Adaptive Distillation for Multimodality Foundation Models

Chen Liang, Jiahui Yu, Ming-Hsuan Yang, Matthew Brown, Yin Cui, Tuo Zhao, Boqing Gong, Tianyi Zhou

专题命中图文多模态：multimodal(abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.CL

2308.12956 2023-08-25 cs.CV cs.AI cs.LG 73%

DLIP: Distilling Language-Image Pre-training

Huafeng Kuang, Jie Wu, Xiawu Zheng, Ming Li, Xuefeng Xiao, Rui Wang, Min Zheng, Rongrong Ji

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

2308.11194 2023-08-23 cs.CV cs.AI 73%

ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data

Maya Varma, Jean-Benoit Delbrouck, Sarah Hooper, Akshay Chaudhari, Curtis Langlotz

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments ICCV 2023

2308.08088 2023-08-17 cs.CV cs.IR cs.MM 73%

Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection

Rui Cao, Ming Shan Hee, Adriel Kuek, Wen-Haw Chong, Roy Ka-Wei Lee, Jing Jiang

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments Camera-ready for 23, ACM MM

2211.12402 2023-08-01 cs.CV cs.CL 73%

X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

Yan Zeng, Xinsong Zhang, Hang Li, Jiawei Wang, Jipeng Zhang, Wangchunshu Zhou

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 25 pages

2209.06794 2023-06-06 cs.CV cs.CL 73%

PaLI: A Jointly-Scaled Multilingual Language-Image Model

Xi Chen, Xiao Wang, Soravit Changpinyo, AJ Piergiovanni, Piotr Padlewski, Daniel Salz, Sebastian Goodman, Adam Grycner, Basil Mustafa, Lucas Beyer, Alexander Kolesnikov, Joan Puigcerver, Nan Ding, Keran Rong, Hassan Akbari, Gaurav Mishra, Linting Xue, Ashish Thapliyal, James Bradbury, Weicheng Kuo, Mojtaba Seyedhosseini, Chao Jia, Burcu Karagol Ayan, Carlos Riquelme, Andreas Steiner, Anelia Angelova, Xiaohua Zhai, Neil Houlsby, Radu Soricut

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments ICLR 2023 (Notable-top-5%)

2306.00890 2023-06-02 cs.CV cs.CL 73%

LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, Jianfeng Gao

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 17 pages; Website: https://aka.ms/llava-med

2302.12971 2023-05-16 cs.CV cs.AI cs.HC 73%

BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding

Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, Nanning Zheng

专题命中图文多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

2212.03640 2023-03-28 cs.CV cs.AI 73%

Fine-tuned CLIP Models are Efficient Video Learners

Hanoona Rasheed, Muhammad Uzair Khattak, Muhammad Maaz, Salman Khan, Fahad Shahbaz Khan

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted at CVPR 2023

2206.07699 2023-02-20 cs.CV cs.CL cs.LG 73%

Write and Paint: Generative Vision-Language Models are Unified Modal Learners

Shizhe Diao, Wangchunshu Zhou, Xinsong Zhang, Jiawei Wang

专题命中图文多模态：multi-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments ICLR 2023

2206.07643 2022-11-21 cs.CV cs.CL cs.LG 73%

Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone

Zi-Yi Dou, Aishwarya Kamath, Zhe Gan, Pengchuan Zhang, Jianfeng Wang, Linjie Li, Zicheng Liu, Ce Liu, Yann LeCun, Nanyun Peng, Jianfeng Gao, Lijuan Wang

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments NeurIPS 2022. Project Website: https://ashkamath.github.io/FIBER_page

2211.00262 2022-11-02 cs.CL cs.CV 73%

Training Vision-Language Models with Less Bimodal Supervision

Elad Segal, Ben Bogin, Jonathan Berant

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments AKBC 2022

2203.11933 2022-10-27 cs.LG cs.CL cs.CV cs.CY 73%

A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning

Hugo Berg, Siobhan Mackenzie Hall, Yash Bhalgat, Wonsuk Yang, Hannah Rose Kirk, Aleksandar Shtedritski, Max Bain

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 17 pages, 4 figures, 7 tables. For code and trained token embeddings, see https://github.com/oxai/debias-vision-lang; Changed to use ACL layout, added joint training with comparison figure, corrected spelling and formatting errors; This paper is accepted for publication at AACL 2022, the official version of record is in the ACL Anthology

2210.12079 2022-10-24 cs.CL cs.CV 73%

Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?

Mitja Nikolaus, Emmanuelle Salin, Stephane Ayache, Abdellah Fourtassi, Benoit Favre

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments To appear at EMNLP 2022

2210.09263 2022-10-18 cs.CV cs.CL 73%

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Zhe Gan, Linjie Li, Chunyuan Li, Lijuan Wang, Zicheng Liu, Jianfeng Gao

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments A survey paper/book on Vision-Language Pre-training (102 pages)

2109.08029 2022-09-14 cs.CV cs.AI 73%

Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering

Ander Salaberria, Gorka Azkune, Oier Lopez de Lacalle, Aitor Soroa, Eneko Agirre

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Under review. 25 pages with 4 figures

Journal ref Expert Systems with Applications, Volume 212, 2023, 118669

2206.01127 2022-09-07 cs.CV cs.CL 73%

VL-BEiT: Generative Vision-Language Pretraining

Hangbo Bao, Wenhui Wang, Li Dong, Furu Wei

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

2207.01056 2022-07-13 cs.CV cs.AI 73%

Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models

Yi Zhang, Junyang Wang, Jitao Sang

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

2205.02655 2022-06-01 cs.CV cs.CL 73%

Language Models Can See: Plugging Visual Controls in Text Generation

Yixuan Su, Tian Lan, Yahui Liu, Fangyu Liu, Dani Yogatama, Yan Wang, Lingpeng Kong, Nigel Collier

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments 21 pages, 5 figures, 5 tables; (v2 adds some experimental details)

2202.07427 2022-05-17 cs.CV cs.CL 73%

On the Complementarity of Images and Text for the Expression of Emotions in Social Media

Anna Khlyzova, Carina Silberer, Roman Klinger

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments WASSA 2022 at ACL 2022, published at https://aclanthology.org/2022.wassa-1.1/ Please cite using https://aclanthology.org/2022.wassa-1.1.bib

2112.02244 2022-04-07 cs.CV cs.CL 73%

LAVT: Language-Aware Vision Transformer for Referring Image Segmentation

Zhao Yang, Jiaqi Wang, Yansong Tang, Kai Chen, Hengshuang Zhao, Philip H. S. Torr

专题命中图文多模态：multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments CVPR 2022

1. 图文多模态 4651 篇

Semi-Supervised Image Captioning Considering Wasserstein Graph Matching

Language-assisted Vision Model Debugger: A Sample-Free Approach to Finding and Fixing Bugs

TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-training

Distinctive Image Captioning: Leveraging Ground Truth Captions in CLIP Guided Reinforcement Learning

PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology

InternLM-XComposer2: Mastering Free-form Text-Image Composition and Comprehension in Vision-Language Large Model

GeoChat: Grounded Large Vision-Language Model for Remote Sensing

VoLTA: Vision-Language Transformer with Weakly-Supervised Local-Feature Alignment

eP-ALM: Efficient Perceptual Augmentation of Language Models

Module-wise Adaptive Distillation for Multimodality Foundation Models

DLIP: Distilling Language-Image Pre-training

ViLLA: Fine-Grained Vision-Language Representation Learning from Real-World Data

Pro-Cap: Leveraging a Frozen Vision-Language Model for Hateful Meme Detection

X$^2$-VLM: All-In-One Pre-trained Model For Vision-Language Tasks

PaLI: A Jointly-Scaled Multilingual Language-Image Model

LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day

BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding

Fine-tuned CLIP Models are Efficient Video Learners

Write and Paint: Generative Vision-Language Models are Unified Modal Learners

Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbone

Training Vision-Language Models with Less Bimodal Supervision

A Prompt Array Keeps the Bias Away: Debiasing Vision-Language Models with Adversarial Learning

Do Vision-and-Language Transformers Learn Grounded Predicate-Noun Dependencies?

Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

Image Captioning for Effective Use of Language Models in Knowledge-Based Visual Question Answering

VL-BEiT: Generative Vision-Language Pretraining

Counterfactually Measuring and Eliminating Social Bias in Vision-Language Pre-training Models

Language Models Can See: Plugging Visual Controls in Text Generation

On the Complementarity of Images and Text for the Expression of Emotions in Social Media

LAVT: Language-Aware Vision Transformer for Referring Image Segmentation