多模态大模型

Unified Loss of Pair Similarity Optimization for Vision-Language Retrieval

Zheng Li, Caili Guo, Xin Wang, Zerun Feng, Jenq-Neng Hwang, Zhongtian Du

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.MM

Comments 16 pages, 5 figures

Translation between Molecules and Natural Language

Carl Edwards, Tuan Lai, Kevin Ros, Garrett Honke, Kyunghyun Cho, Heng Ji

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CL、cs.AI

Comments Accepted at EMNLP 2022. Data and code can be found on [Github](https://github.com/blender-nlp/MolT5)

TANGO: Text-driven Photorealistic and Robust 3D Stylization via Lighting Decomposition

Yongwei Chen, Rui Chen, Jiabao Lei, Yabin Zhang, Kui Jia

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by NeurIPS 2022

On-the-fly Object Detection using StyleGAN with CLIP Guidance

Yuzhe Lu, Shusen Liu, Jayaraman J. Thiagarajan, Wesam Sakla, Rushil Anirudh

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.AI

Aligning MAGMA by Few-Shot Learning and Finetuning

Jean-Charles Layoun, Alexis Roger, Irina Rish

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI

Comments Accepted by the Montreal AI Symposium conference in 2022

On Advances in Text Generation from Images Beyond Captioning: A Case Study in Self-Rationalization

Shruti Palaskar, Akshita Bhagia, Yonatan Bisk, Florian Metze, Alan W Black, Ana Marasović

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments v2: EMNLP Findings 2022 accepted paper camera-ready version. 9 pages main, 2 pages appendix

Distilled Dual-Encoder Model for Vision-Language Understanding

Zekun Wang, Wenhui Wang, Haichao Zhu, Ming Liu, Bing Qin, Furu Wei

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments EMNLP 2022

EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning

Tiannan Wang, Wangchunshu Zhou, Yan Zeng, Xinsong Zhang

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments work in progress

A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibility

Andrea Burns, Deniz Arsan, Sanjna Agrawal, Ranjitha Kumar, Kate Saenko, Bryan A. Plummer

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Accepted at the European Conference on Computer Vision (ECCV) 2022. This is a new version of the paper with additional experimental results and a few prior implementation bugs fixed

A Survey of Vision-Language Pre-Trained Models

Yifan Du, Zikang Liu, Junyi Li, Wayne Xin Zhao

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Accepted by IJCAI-2022 survey track

Contrastive Vision-Language Pre-training with Limited Resources

Quan Cui, Boyan Zhou, Yu Guo, Weidong Yin, Hao Wu, Osamu Yoshie, Yubo Chen

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.MM

Comments Accepted to ECCV2022

cViL: Cross-Lingual Training of Vision-Language Models using Knowledge Distillation

Kshitij Gupta, Devansh Gautam, Radhika Mamidi

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted at ICPR 2022; 9 pages

VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Experts

Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Furu Wei

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments Work in progress

CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models

Yuan Yao, Ao Zhang, Zhengyan Zhang, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Work in progress

A Frustratingly Simple Approach for End-to-End Image Captioning

Ziyang Luo, Yadong Xi, Rongsheng Zhang, Jing Ma

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Work in progress

Interactive Audio-text Representation for Automated Audio Captioning with Contrastive Learning

Chen Chen, Nana Hou, Yuchen Hu, Heqing Zou, Xiaofeng Qi, Eng Siong Chng

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CL、eess.AS

Comments Submitted to Interspeech 2022

Multi-View Transformer for 3D Visual Grounding

Shijia Huang, Yilun Chen, Jiaya Jia, Liwei Wang

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

Comments cvpr2022

Scaling Up Vision-Language Pre-training for Image Captioning

Xiaowei Hu, Zhe Gan, Jianfeng Wang, Zhengyuan Yang, Zicheng Liu, Yumao Lu, Lijuan Wang

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

DenseCLIP: Language-Guided Dense Prediction with Context-Aware Prompting

Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, Jiwen Lu

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted to CVPR2022. Project page: https://denseclip.ivg-research.xyz

An Empirical Study of Training End-to-End Vision-and-Language Transformers

Zi-Yi Dou, Yichong Xu, Zhe Gan, Jianfeng Wang, Shuohang Wang, Lijuan Wang, Chenguang Zhu, Pengchuan Zhang, Lu Yuan, Nanyun Peng, Zicheng Liu, Michael Zeng

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments CVPR 2022

UNIMO-2: End-to-End Unified Vision-Language Grounded Learning

Wei Li, Can Gao, Guocheng Niu, Xinyan Xiao, Hao Liu, Jiachen Liu, Hua Wu, Haifeng Wang

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

Comments Accepted by ACL2022

StyleBabel: Artistic Style Tagging and Captioning

Dan Ruta, Andrew Gilbert, Pranav Aggarwal, Naveen Marri, Ajinkya Kale, Jo Briggs, Chris Speed, Hailin Jin, Baldo Faieta, Alex Filipkowski, Zhe Lin, John Collomosse

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL

StyleCLIPDraw: Coupling Content and Style in Text-to-Drawing Synthesis

Peter Schaldenbrand, Zhixuan Liu, Jean Oh

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments Superseded by arXiv:2202.12362

ERNIE-ViLG: Unified Generative Pre-training for Bidirectional Vision-Language Generation

Han Zhang, Weichong Yin, Yewei Fang, Lanxin Li, Boqiang Duan, Zhihua Wu, Yu Sun, Hao Tian, Hua Wu, Haifeng Wang

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments 15 pages, 7 figures

Playing Lottery Tickets with Vision and Language

Zhe Gan, Yen-Chun Chen, Linjie Li, Tianlong Chen, Yu Cheng, Shuohang Wang, Jingjing Liu, Lijuan Wang, Zicheng Liu

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted to AAAI 2022

PointCLIP: Point Cloud Understanding by CLIP

Renrui Zhang, Ziyu Guo, Wei Zhang, Kunchang Li, Xupeng Miao, Bin Cui, Yu Qiao, Peng Gao, Hongsheng Li

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.AI

Comments Open sourced, Code and Model Available

From Show to Tell: A Survey on Deep Learning-based Image Captioning

Matteo Stefanini, Marcella Cornia, Lorenzo Baraldi, Silvia Cascianelli, Giuseppe Fiameni, Rita Cucchiara

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL

CL4AC: A Contrastive Loss for Audio Captioning

Xubo Liu, Qiushi Huang, Xinhao Mei, Tom Ko, H Lilian Tang, Mark D. Plumbley, Wenwu Wang

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.AI、eess.AS

Comments The first two authors contributed equally, 5 pages, 3 figures, accepted by DCASE2021 Workshop

Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling

Renrui Zhang, Rongyao Fang, Wei Zhang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, Hongsheng Li

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL

Comments preprints

Understanding Guided Image Captioning Performance across Domains

Edwin G. Ng, Bo Pang, Piyush Sharma, Radu Soricut

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL

Comments Proceedings of CoNLL 2021