多模态大模型

Unified Contrastive Learning in Image-Text-Label Space

Jianwei Yang, Chunyuan Li, Pengchuan Zhang, Bin Xiao, Ce Liu, Lu Yuan, Jianfeng Gao

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.AI

Comments CVPR 2022

Assessing Multilingual Fairness in Pre-trained Multimodal Representations

Jialu Wang, Yang Liu, Xin Eric Wang

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CL、cs.AI

Comments 15 pages, 18 figures

Two-stream Hierarchical Similarity Reasoning for Image-text Matching

Ran Chen, Hanli Wang, Lei Wang, Sam Kwong

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.MM

MAGIC: Multimodal relAtional Graph adversarIal inferenCe for Diverse and Unpaired Text-based Image Captioning

Wenqiao Zhang, Haochen Shi, Jiannan Guo, Shengyu Zhang, Qingpeng Cai, Juncheng Li, Sihui Luo, Yueting Zhuang

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Journal ref AAAI 2022

RedCaps: web-curated image-text data created by the people, for the people

Karan Desai, Gaurav Kaul, Zubin Aysola, Justin Johnson

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.CL

Comments NeurIPS 2021 Datasets and Benchmarks. Website: https://redcaps.xyz

Negative Sample is Negative in Its Own Way: Tailoring Negative Sentences for Image-Text Retrieval

Zhihao Fan, Zhongyu Wei, Zejun Li, Siyuan Wang, Jianqing Fan

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.CL

Exploiting Cross-Modal Prediction and Relation Consistency for Semi-Supervised Image Captioning

Yang Yang, Hongchen Wei, Hengshu Zhu, Dianhai Yu, Hui Xiong, Jian Yang

专题命中 图文多模态 ：cross-modal(title,abstract)；分类 cs.CV、cs.AI

Is An Image Worth Five Sentences? A New Look into Semantics for Image-Text Matching

Ali Furkan Biten, Andres Mafla, Lluis Gomez, Dimosthenis Karatzas

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted WACV 2022

Hierarchical Multimodal Transformer to Summarize Videos

Bin Zhao, Maoguo Gong, Xuelong Li

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Multilingual Multimodal Pre-training for Zero-Shot Cross-Lingual Transfer of Vision-Language Models

Po-Yao Huang, Mandela Patrick, Junjie Hu, Graham Neubig, Florian Metze, Alexander Hauptmann

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments accepted by NAACL 2021

Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Concepts

Soravit Changpinyo, Piyush Sharma, Nan Ding, Radu Soricut

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.CL

Comments IEEE Conference on Computer Vision and Pattern Recognition (CVPR 2021). Our dataset is available at https://github.com/google-research-datasets/conceptual-12m

Confidence-aware Non-repetitive Multimodal Transformers for TextCaps

Zhaokai Wang, Renda Bao, Qi Wu, Si Liu

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 9 pages; Accepted by AAAI 2021

Reading Isn't Believing: Adversarial Attacks On Multi-Modal Neurons

David A. Noever, Samantha E. Miller Noever

专题命中 图文多模态 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Similarity Reasoning and Filtration for Image-Text Matching

Haiwen Diao, Ying Zhang, Lin Ma, Huchuan Lu

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.MM

Comments 14 pages, 8 figures, Accepted by AAAI2021

Multimodal Learning for Hateful Memes Detection

Yi Zhou, Zhenhao Chen

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.AI

Multimodal Pretraining for Dense Video Captioning

Gabriel Huang, Bo Pang, Zhenhai Zhu, Clara Rivera, Radu Soricut

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments AACL-IJCNLP 2020

Unsupervised Multimodal Neural Machine Translation with Pseudo Visual Pivoting

Po-Yao Huang, Junjie Hu, Xiaojun Chang, Alexander Hauptmann

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by ACL 2020

More Grounded Image Captioning by Distilling Image-Text Matching Model

Yuanen Zhou, Meng Wang, Daqing Liu, Zhenzhen Hu, Hanwang Zhang

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by CVPR 2020

Vision-Dialog Navigation by Exploring Cross-modal Memory

Yi Zhu, Fengda Zhu, Zhaohuan Zhan, Bingqian Lin, Jianbin Jiao, Xiaojun Chang, Xiaodan Liang

专题命中 图文多模态 ：cross-modal(title,abstract)；分类 cs.CV、cs.CL

Comments CVPR2020

Exploring and Distilling Cross-Modal Information for Image Captioning

Fenglin Liu, Xuancheng Ren, Yuanxin Liu, Kai Lei, Xu Sun

专题命中 图文多模态 ：cross-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted by IJCAI 2019

Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling

Hao Zhang, Bo Chen, Long Tian, Zhengjue Wang, Mingyuan Zhou

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.CL

Comments ICLR 2020

Low-Rank HOCA: Efficient High-Order Cross-Modal Attention for Video Captioning

Tao Jin, Siyu Huang, Yingming Li, Zhongfei Zhang

专题命中 图文多模态 ：cross-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted as a long paper at EMNLP 2019

Learning Visual Relation Priors for Image-Text Matching and Image Captioning with Neural Scene Graph Generators

Kuang-Huei Lee, Hamid Palangi, Xi Chen, Houdong Hu, Jianfeng Gao

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.AI

Multi-modal Discriminative Model for Vision-and-Language Navigation

Haoshuo Huang, Vihan Jain, Harsh Mehta, Jason Baldridge, Eugene Ie

专题命中 图文多模态 ：multi-modal(title,abstract)；分类 cs.CV、cs.CL

Comments Accepted at SpLU-RoboNLP 2019 (workshop at NAACL)

Interactive-predictive neural multimodal systems

Álvaro Peris, Francisco Casacuberta

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments To appear at IbPRIA 2019

Stacked Cross Attention for Image-Text Matching

Kuang-Huei Lee, Xi Chen, Gang Hua, Houdong Hu, Xiaodong He

专题命中 图文多模态 ：image-text(title,abstract)；分类 cs.CV、cs.AI

Comments Accepted to ECCV 2018

Quantifying the visual concreteness of words and topics in multimodal datasets

Jack Hessel, David Mimno, Lillian Lee

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments NAACL HLT 2018, 14 pages, 6 figures, data available at http://www.cs.cornell.edu/~jhessel/concreteness/concreteness.html

Journal ref 2018 North American Chapter of the Association for Computational Linguistics: Human Language Technologies (NAACL HLT)

Self-Guiding Multimodal LSTM - when we do not have a perfect training dataset for image captioning

Yang Xian, Yingli Tian

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments The paper is under consideration at Computer Vision and Image Understanding

Image Pivoting for Learning Multilingual Multimodal Representations

Spandana Gella, Rico Sennrich, Frank Keller, Mirella Lapata

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments 7 pages, EMNLP 2017

Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN)

Junhua Mao, Wei Xu, Yi Yang, Jiang Wang, Zhiheng Huang, Alan Yuille

专题命中 图文多模态 ：multimodal(title,abstract)；分类 cs.CV、cs.CL

Comments Add a simple strategy to boost the performance of image captioning task significantly. More details are shown in Section 8 of the paper. The code and related data are available at https://github.com/mjhucla/mRNN-CR ;. arXiv admin note: substantial text overlap with arXiv:1410.1090

Journal ref ICLR 2015