多模态大模型

Video Question Answering Using CLIP-Guided Visual-Text Attention

Shuhong Ye, Weikai Kong, Chenglin Yao, Jianfeng Ren, Xudong Jiang

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Submitted to the 2023 IEEE International Conference on Image Processing (ICIP 2023)

IPA-CLIP: Integrating Phonetic Priors into Vision and Language Pretraining

Chihaya Matsuhira, Marc A. Kastner, Takahiro Komamizu, Takatsugu Hirayama, Keisuke Doman, Yasutomo Kawanishi, Ichiro Ide

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CL、cs.MM、eess.AS

Comments 11 pages, 8 figures, 5 Tables

DeCap: Decoding CLIP Latents for Zero-Shot Captioning via Text-Only Training

Wei Li, Linchao Zhu, Longyin Wen, Yi Yang

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by ICLR 2023. Code is available at https://github.com/dhg-wei/DeCap

Open-domain Visual Entity Recognition: Towards Recognizing Millions of Wikipedia Entities

Hexiang Hu, Yi Luan, Yang Chen, Urvashi Khandelwal, Mandar Joshi, Kenton Lee, Kristina Toutanova, Ming-Wei Chang

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Dataset available at https://open-vision-language.github.io/oven

Paparazzi: A Deep Dive into the Capabilities of Language and Vision Models for Grounding Viewpoint Descriptions

Henrik Voigt, Jan Hombeck, Monique Meuschke, Kai Lawonn, Sina Zarrieß

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

VAuLT: Augmenting the Vision-and-Language Transformer for Sentiment Classification on Social Media

Georgios Chochlakis, Tejas Srinivasan, Jesse Thomason, Shrikanth Narayanan

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 5 pages, 1 figure

Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains

Pierre Chambon, Christian Bluethgen, Curtis P. Langlotz, Akshay Chaudhari

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 17 pages, 8 figures

Journal ref Foundation Models for Decision Making Workshop at Neural Information Processing Systems, 2022

Semantic-Conditional Diffusion Networks for Image Captioning

Jianjie Luo, Yehao Li, Yingwei Pan, Ting Yao, Jianlin Feng, Hongyang Chao, Tao Mei

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Source code is available at \url{https://github.com/YehLi/xmodaler/tree/master/configs/image_caption/scdnet}

PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models

Yuan Yao, Qianyu Chen, Ao Zhang, Wei Ji, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by EMNLP 2022

CPL: Counterfactual Prompt Learning for Vision and Language Models

Xuehai He, Diji Yang, Weixi Feng, Tsu-Jui Fu, Arjun Akula, Varun Jampani, Pradyumna Narayana, Sugato Basu, William Yang Wang, Xin Eric Wang

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

TVLT: Textless Vision-Language Transformer

Zineng Tang, Jaemin Cho, Yixin Nie, Mohit Bansal

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments NeurIPS 2022 Oral (21 pages; the first three authors contributed equally)

Towards Adversarial Attack on Vision-Language Pre-training Models

Jiaming Zhang, Qi Yi, Jitao Sang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted by ACM MM2022. Code is available in GitHub

Visual Clues: Bridging Vision and Language Foundations for Image Paragraph Captioning

Yujia Xie, Luowei Zhou, Xiyang Dai, Lu Yuan, Nguyen Bach, Ce Liu, Michael Zeng

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

VL-Taboo: An Analysis of Attribute-based Zero-shot Capabilities of Vision-Language Models

Felix Vogel, Nina Shvetsova, Leonid Karlinsky, Hilde Kuehne

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Curriculum Learning for Data-Efficient Vision-Language Alignment

Tejas Srinivasan, Xiang Ren, Jesse Thomason

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification

Renrui Zhang, Zhang Wei, Rongyao Fang, Peng Gao, Kunchang Li, Jifeng Dai, Yu Qiao, Hongsheng Li

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by ECCV 2022. arXiv admin note: substantial text overlap with arXiv:2111.03930

Markedness in Visual Semantic AI

Robert Wolfe, Aylin Caliskan

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments To be published at ACM FAccT 2022

Learning Audio-Video Modalities from Image Captions

Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, Cordelia Schmid

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.MM、eess.AS

Vision-Language Intelligence: Tasks, Representation Learning, and Large Models

Feng Li, Hao Zhang, Yi-Fan Zhang, Shilong Liu, Jian Guo, Lionel M. Ni, PengChuan Zhang, Lei Zhang

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Uni-EDEN: Universal Encoder-Decoder Network by Multi-Granular Vision-Language Pre-training

Yehao Li, Jiahao Fan, Yingwei Pan, Ting Yao, Weiyao Lin, Tao Mei

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM)

CLIP-It! Language-Guided Video Summarization

Medhini Narasimhan, Anna Rohrbach, Trevor Darrell

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.AI、cs.MM

Comments Neurips 2021. Website at https://medhini.github.io/clip_it/

Journal ref Thirty-Fifth Conference on Neural Information Processing Systems. 2021

Does language help generalization in vision models?

Benjamin Devillers, Bhavin Choksi, Romain Bielawski, Rufin VanRullen

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Paper accepted at the CoNLL 2021 conference. This version: section added on the performance of the visual and visio-linguistic models on linquistic tasks

Similar Scenes arouse Similar Emotions: Parallel Data Augmentation for Stylized Image Captioning

Guodun Li, Yuchen Zhai, Zehao Lin, Yin Zhang

专题命中 图文多模态 ：multi-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

Comments Accepted at ACM Multimedia (ACMMM) 2021

Grid-VLP: Revisiting Grid Features for Vision-Language Pre-training

Ming Yan, Haiyang Xu, Chenliang Li, Bin Bi, Junfeng Tian, Min Gui, Wei Wang

专题命中 图文多模态 ：cross-modal(abstract)；分类 cs.CV、cs.CL、cs.MM

Exploring the Limits of Out-of-Distribution Detection

Stanislav Fort, Jie Ren, Balaji Lakshminarayanan

专题命中 图文多模态 ：multi-modal(abstract)；image-text(abstract)

Comments S.F. and J.R. contributed equally

FixMyPose: Pose Correctional Captioning and Retrieval

Hyounghun Kim, Abhay Zala, Graham Burri, Mohit Bansal

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments AAAI 2021 (18 pages, 16 figures; webpage: https://fixmypose-unc.github.io/)

Vokenization: Improving Language Understanding with Contextualized, Visual-Grounded Supervision

Hao Tan, Mohit Bansal

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments EMNLP 2020 (15 pages)

The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

Jiayuan Mao, Chuang Gan, Pushmeet Kohli, Joshua B. Tenenbaum, Jiajun Wu

专题命中 图文多模态 ：image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ICLR 2019 (Oral). Project page: http://nscl.csail.mit.edu/

Bilinear Attention Networks

Jin-Hwa Kim, Jaehyun Jun, Byoung-Tak Zhang

专题命中 图文多模态 ：multimodal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by NIPS 2018; Figure 1 was updated

CLIP-CC-Bench: Evaluating Paragraph-Level Video Descriptions in Video-Language Models

CLIP-CC-Bench：评估视频语言模型中的段落级视频描述

Mukhtiar Ali, Harsh Dubey, Sugam Mishra, Chulwoo Pack

机构 * South Dakota State University（南达科他州立大学） 纠错

专题命中 图文多模态 ：multimodal(abstract,comments)；分类 cs.CV、cs.MM

AI总结 CLIP-CC-Bench是针对视频语言模型的长篇段落级视频描述评估套件，采用多LLM嵌入模型集成与粗细粒度语义匹配方法，评估17种模型填补了现有短片段基准的空白。

Comments Accepted and presented at EvalMG 2026, the Second Workshop on Evaluation for Multimodal Generation, co-located with ACM SIGIR 2026