多模态大模型

UrbanCross: Enhancing Satellite Image-Text Retrieval with Cross-Domain Adaptation

Siru Zhong, Xixuan Hao, Yibo Yan, Ying Zhang, Yangqiu Song, Yuxuan Liang

专题命中 图文多模态 ：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Brandon McKinzie, Zhe Gan, Jean-Philippe Fauconnier, Sam Dodge, Bowen Zhang, Philipp Dufter, Dhruti Shah, Xianzhi Du, Futang Peng, Floris Weers, Anton Belyi, Haotian Zhang, Karanjeet Singh, Doug Kang, Ankur Jain, Hongyu Hè, Max Schwarzer, Tom Gunter, Xiang Kong, Aonan Zhang, Jianyu Wang, Chong Wang, Nan Du, Tao Lei, Sam Wiseman, Guoli Yin, Mark Lee, Zirui Wang, Ruoming Pang, Peter Grasch, Alexander Toshev, Yinfei Yang

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning

Siteng Huang, Biao Gong, Yutong Feng, Min Zhang, Yiliang Lv, Donglin Wang

专题命中 图文多模态 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

Comments CVPR 2024

Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities

Yiyuan Zhang, Xiaohan Ding, Kaixiong Gong, Yixiao Ge, Ying Shan, Xiangyu Yue

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

Comments CVPR 2024. Code and models are available at https://github.com/AILab-CVC/M2PT

DreamLLM: Synergistic Multimodal Comprehension and Creation

Runpei Dong, Chunrui Han, Yuang Peng, Zekun Qi, Zheng Ge, Jinrong Yang, Liang Zhao, Jianjian Sun, Hongyu Zhou, Haoran Wei, Xiangwen Kong, Xiangyu Zhang, Kaisheng Ma, Li Yi

专题命中 图文多模态 ：multimodal(title,abstract)；MLLM(abstract)；分类 cs.CV、cs.CL

Comments ICLR 2024 (Spotlight)

RED-DOT: Multimodal Fact-checking via Relevant Evidence Detection

Stefanos-Iordanis Papadopoulos, Christos Koutlis, Symeon Papadopoulos, Panagiotis C. Petrantonakis

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.MM

Improving fine-grained understanding in image-text pre-training

Ioana Bica, Anastasija Ilić, Matthias Bauer, Goker Erdogan, Matko Bošnjak, Christos Kaplanis, Alexey A. Gritsenko, Matthias Minderer, Charles Blundell, Razvan Pascanu, Jovana Mitrović

专题命中 图文多模态 ：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.AI

Comments 26 pages

Training Transitive and Commutative Multimodal Transformers with LoReTTa

Manuel Tran, Yashin Dicente Cid, Amal Lahiani, Fabian J. Theis, Tingying Peng, Eldad Klaiman

专题命中 图文多模态 ：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CV、cs.AI

Comments Accepted at NeurIPS 2023 (poster). Camera-ready version

Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models

Xin He, Longhui Wei, Lingxi Xie, Qi Tian

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.MM

CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare

Akash Ghosh, Arkadeep Acharya, Raghav Jain, Sriparna Saha, Aman Chadha, Setu Sinha

专题命中 图文多模态 ：multimodal(title,abstract)；multimodal foundation model(abstract)；分类 cs.CL、cs.AI

Comments AAAI 2024

SoftCLIP: Softer Cross-modal Alignment Makes CLIP Stronger

Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu Enwei Zhang, Wei Liu, Jie Yang, Ke Li, Xing Sun

专题命中 图文多模态 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Improving Cross-modal Alignment with Synthetic Pairs for Text-only Image Captioning

Zhiyue Liu, Jinyuan Liu, Fanrong Ma

专题命中 图文多模态 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments AAAI2024

Delving into Multimodal Prompting for Fine-grained Visual Classification

Xin Jiang, Hao Tang, Junyao Gao, Xiaoyu Du, Shengfeng He, Zechao Li

专题命中 图文多模态 ：multimodal(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments This paper is accepted by AAAI 2024

ASIF: Coupled Data Turns Unimodal Models to Multimodal Without Training

Antonio Norelli, Marco Fumero, Valentino Maiorca, Luca Moschella, Emanuele Rodolà, Francesco Locatello

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments 17 pages

COCO-Counterfactuals: Automatically Constructed Counterfactual Examples for Image-Text Pairs

Tiep Le, Vasudev Lal, Phillip Howard

专题命中 图文多模态 ：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

Comments Accepted to NeurIPS 2023 Datasets and Benchmarks Track

A Suite of Generative Tasks for Multi-Level Multimodal Webpage Understanding

Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, Mandy Guo

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted in EMNLP 2023, revision contains camera ready edits. Data can be downloaded at https://github.com/google-research-datasets/wit/blob/main/wikiweb2m.md

Aligning Large Multimodal Models with Factually Augmented RLHF

Zhiqing Sun, Sheng Shen, Shengcao Cao, Haotian Liu, Chunyuan Li, Yikang Shen, Chuang Gan, Liang-Yan Gui, Yu-Xiong Wang, Yiming Yang, Kurt Keutzer, Trevor Darrell

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Preprint

MaMMUT: A Simple Architecture for Joint Learning for MultiModal Tasks

Weicheng Kuo, AJ Piergiovanni, Dahun Kim, Xiyang Luo, Ben Caine, Wei Li, Abhijit Ogale, Luowei Zhou, Andrew Dai, Zhifeng Chen, Claire Cui, Anelia Angelova

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Published in Transactions on Machine Learning Research ( https://jmlr.org/tmlr/ ). 18 pages, 4 figures

Med-Flamingo: a Multimodal Medical Few-shot Learner

Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Cyril Zakka, Yash Dalmia, Eduardo Pontes Reis, Pranav Rajpurkar, Jure Leskovec

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Preprint

Target-oriented Sentiment Classification with Sequential Cross-modal Semantic Graph

Yufeng Huang, Zhuo Chen, Jiaoyan Chen, Jeff Z. Pan, Zhen Yao, Wen Zhang

专题命中 图文多模态 ：cross-modal(title,abstract)；multi-modal(abstract)；分类 cs.CV、cs.AI

Comments ICANN 2023, https://github.com/zjukg/SeqCSG

mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connections

Chenliang Li, Haiyang Xu, Junfeng Tian, Wei Wang, Ming Yan, Bin Bi, Jiabo Ye, Hehong Chen, Guohai Xu, Zheng Cao, Ji Zhang, Songfang Huang, Fei Huang, Jingren Zhou, Luo Si

专题命中 图文多模态 ：cross-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Journal ref EMNLP2022

Cross-modal Attention Congruence Regularization for Vision-Language Relation Alignment

Rohan Pandey, Rulin Shao, Paul Pu Liang, Ruslan Salakhutdinov, Louis-Philippe Morency

专题命中 图文多模态 ：cross-modal(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.CL

Comments ACL 2023

Switch-BERT: Learning to Model Multimodal Interactions by Switching Attention and Input

Qingpei Guo, Kaisheng Yao, Wei Chu

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.AI

Comments Accepted by ECCV2022

Exploring Better Text Image Translation with Multimodal Codebook

Zhibin Lan, Jiawei Yu, Xiang Li, Wen Zhang, Jian Luan, Bin Wang, Degen Huang, Jinsong Su

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CL、cs.AI

Comments Accepted by ACL 2023 Main Conference

WikiWeb2M: A Page-Level Multimodal Wikipedia Dataset

Andrea Burns, Krishna Srinivasan, Joshua Ainslie, Geoff Brown, Bryan A. Plummer, Kate Saenko, Jianmo Ni, Mandy Guo

专题命中 图文多模态 ：multimodal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Accepted at the WikiWorkshop 2023. Data is readily available at https://github.com/google-research-datasets/wit/blob/main/wikiweb2m.md. arXiv admin note: text overlap with arXiv:2305.03668

Refined Vision-Language Modeling for Fine-grained Multi-modal Pre-training

Lisai Zhang, Qingcai Chen, Zhijian Chen, Yunpeng Han, Zhonghua Li, Zhao Cao

专题命中 图文多模态 ：multi-modal(title,abstract)；image-text(abstract)；分类 cs.CV、cs.CL

Comments Work in progress, v0.2

Plug-and-Play Regulators for Image-Text Matching

Haiwen Diao, Ying Zhang, Wei Liu, Xiang Ruan, Huchuan Lu

专题命中 图文多模态 ：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

Comments 13 pages, 9 figures, Accepted by TIP2023

CoBIT: A Contrastive Bi-directional Image-Text Generation Model

Haoxuan You, Mandy Guo, Zhecan Wang, Kai-Wei Chang, Jason Baldridge, Jiahui Yu

专题命中 图文多模态 ：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.CL

Comments 14 pages, 5 figures

Scene Graph Based Fusion Network For Image-Text Retrieval

Guoliang Wang, Yanlei Shang, Yong Chen

专题命中 图文多模态 ：image-text(title,abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

HGAN: Hierarchical Graph Alignment Network for Image-Text Retrieval

Jie Guo, Meiting Wang, Yan Zhou, Bin Song, Yuhao Chi, Wei Fan, Jianglong Chang

专题命中 图文多模态 ：image-text(title,abstract)；multimodal(abstract)；分类 cs.CV、cs.MM