多模态大模型 - arXivDaily 专题

2511.22341 2025-12-01 cs.CV cs.LG 77%

Unexplored flaws in multiple-choice VQA evaluations

多选式视觉问答评估中的未探索缺陷

Fabio Rosenthal, Sebastian Schmidt, Thorsten Graf, Thorsten Bagodonat, Stephan Günnemann, Leo Schwinn

机构 * Technical University of Munich（慕尼黑技术大学）； Volkswagen AG（大众集团）； Munich Data Science Institute（慕尼黑数据科学研究所）

专题命中图文多模态：multimodal(abstract)；MLLM(abstract)；image-text(abstract)；分类 cs.CV

AI总结本文揭示了多选式视觉问答评估中因提示格式变化导致的未探索偏差，指出其对MLLM评估结果的显著影响，并表明现有缓解策略无法有效应对这些新发现的偏见。

2503.17551 2025-10-03 cs.MM cs.AI cs.CV cs.SD eess.AS 77%

Audio-Enhanced Vision-Language Modeling with Latent Space Broadening for High Quality Data Expansion

Yu Sun, Yin Li, Ruixiao Sun, Chunhui Liu, Fangming Zhou, Ze Jin, Linjie Wang, Xiang Shen, Zhuolin Hao, Hongyu Xiong

机构 * ByteDance Inc.（字节跳动公司）

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI、cs.MM

2502.16680 2025-09-03 cs.CV 77%

AeroReformer: Aerial Referring Transformer for UAV-based Referring Image Segmentation

Rui Li, Xiaowei Zhao

机构 * Intelligent Control \& Smart Energy (ICSE) Research Group, School of Engineering, University of Warwick, Coventry, CV4 7AL, UK

专题命中图文多模态：multimodal(abstract)；MLLM(abstract)；cross-modal(abstract)；分类 cs.CV

2507.21489 2025-07-30 cs.CV 77%

Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrieval

Zhichuan Wang, Yang Zhou, Zhe Liu, Rui Yu, Song Bai, Yulong Wang, Xinwei He, Xiang Bai

机构 * Huazhong Agricultural University（华中农业大学）； Shenzhen University（深圳大学）； The University of Hong Kong（香港大学）； University of Louisville（路易斯安那大学）； ByteDance（字节跳动）； Huazhong University of Science and Technology（华中科技大学）

专题命中图文多模态：multi-modal(abstract)；MLLM(abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted to ICCV 2025

2506.12430 2025-07-14 cs.CR cs.CV 77%

Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025

Zonghao Ying, Siyang Wu, Run Hao, Peng Ying, Shixuan Sun, Pengyu Chen, Junze Chen, Hao Du, Kaiwen Shen, Shangkun Wu, Jiwei Wei, Shiyuan He, Yang Yang, Xiaohai Xu, Ke Ma, Qianqian Xu, Qingming Huang, Shi Lin, Xun Wang, Changting Lin, Meng Han, Yilei Jiang, Siqi Lai, Yaozhi Zheng, Yifei Song, Xiangyu Yue, Zonglei Jing, Tianyuan Zhang, Zhilei Zhu, Aishan Liu, Jiakai Wang, Siyuan Liang, Xianglong Kong, Hainan Li, Junjie Mu, Haotong Qin, Yue Yu, Lei Chen, Felix Juefei-Xu, Qing Guo, Xinyun Chen, Yew Soon Ong, Xianglong Liu, Dawn Song, Alan Yuille, Philip Torr, Dacheng Tao

机构 * Beihang University（北航）； Zhongguancun Laboratory（中关村实验室）； Hefei Comprehensive National Science Center（合肥综合性国家科学中心）； ETH Zürich（苏黎世联邦理工学院）； Pengcheng Laboratory（鹏城实验室）； Tsinghua University（清华大学）； University of California, Berkeley（加州大学伯克利分校）； Johns Hopkins University（约翰霍普金斯大学）； University of Oxford（牛津大学）； Meta ； Google Brain（谷歌脑）； Nanyang Technological University（南洋理工大学）； Aarhus University（哥本哈根大学）； China University of Mining（中国矿业大学）； University of Electronic Science（电子科学大学）； School of Electronic, Electrical（电子、电气与通信工程学院）； Key Laboratory of Intelligent Information Processing, Institute of Computing Technology, CAS（智能信息处理国家重点实验室）； School of Computer Science（计算机科学学院）； Key Laboratory of Big Data Mining（大数据挖掘国家重点实验室）； Zhejiang Gongshang University（浙江工商大学）； Binjiang Institute of Zhejiang University（浙江大学滨江学院）； Zhejiang University（浙江大学）； The Chinese University of Hong Kong（香港中文大学）

专题命中图文多模态：multimodal(abstract)；MLLM(abstract)；image-text(abstract)；分类 cs.CV

Comments AdvML@CVPR Challenge Report

2505.22396 2025-05-29 cs.CV 77%

Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs

Xudong Li, Mengdan Zhang, Peixian Chen, Xiawu Zheng, Yan Zhang, Jingyuan Zheng, Yunhang Shen, Ke Li, Chaoyou Fu, Xing Sun, Rongrong Ji

机构 * Key Laboratory of Multimedia Trusted Perception and Efficient Computing, Ministry of Education of China, Xiamen University（中国教育部多媒体可信感知与高效计算重点实验室，厦门大学）； Tencent Youtu Lab（腾讯优图实验室）； Nanjing University（南京大学）

专题命中图文多模态：multimodal(abstract)；multi-modal(abstract)；cross-modal(abstract)；分类 cs.CV

2505.09372 2025-05-15 cs.CV 77%

MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessment

Siyuan Yan, Xieji Li, Ming Hu, Yiwen Jiang, Zhen Yu, Zongyuan Ge

机构 * Faculty of Engineering, Monash University, Melbourne, Australia（墨尔本大学工程学院）； AIM for Health Lab, Monash University, Victoria, Australia（墨尔本大学健康人工智能实验室）

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments MICCAI2025 early acceptance; First two authors contribute equally

2505.02448 2025-05-06 cs.CV 77%

Recent Advances in Out-of-Distribution Detection with CLIP-Like Models: A Survey

Chaohua Li, Enhao Zhang, Chuanxing Geng, Songcan Chen

机构 * College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics（南京航空航天大学计算机科学与技术学院）； MIIT Key Laboratory of Pattern Analysis and Machine Intelligence（信息产业部模式分析与机器智能重点实验室）； Department of Computer Science, Hong Kong Baptist University（香港 Baptist 大学计算机科学系）

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

2504.21040 2025-05-01 cs.CV 77%

Can a Large Language Model Assess Urban Design Quality? Evaluating Walkability Metrics Across Expertise Levels

Chenyi Cai, Kosuke Kuriyama, Youlong Gu, Filip Biljecki, Pieter Herthogs

机构 * Singapore-ETH Centre, Future Cities Lab Global Programme, CREATE campus（新加坡-ETH中心，未来城市实验室全球计划，CREATE校区）； National University of Singapore（新加坡国立大学）； Takenaka Corporation（Takenaka公司）； Department of Architecture, National University of Singapore（新加坡国立大学建筑系）

专题命中图文多模态：multimodal(abstract)；MLLM(abstract)；image-text(abstract)；分类 cs.CV

2504.10462 2025-04-15 cs.CV 77%

The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer

Weixian Lei, Jiacong Wang, Haochen Wang, Xiangtai Li, Jun Hao Liew, Jiashi Feng, Zilong Huang

专题命中图文多模态：multimodal(abstract)；MLLM(abstract)；cross-modal(abstract)；分类 cs.CV

2503.12127 2025-03-18 cs.CV cs.AI cs.CL cs.MM 77%

Hyperbolic Safety-Aware Vision-Language Models

Tobia Poppi, Tejaswi Kasarla, Pascal Mettes, Lorenzo Baraldi, Rita Cucchiara

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments CVPR 2025

2408.12928 2025-03-17 cs.CV 77%

ParGo: Bridging Vision-Language with Partial and Global Views

An-Lan Wang, Bin Shan, Wei Shi, Kun-Yu Lin, Xiang Fei, Guozhi Tang, Lei Liao, Can Huang, Jingqun Tang, Wei-Shi Zheng

专题命中图文多模态：multimodal(abstract)；MLLM(abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted by AAAI 2025

2312.03700 2025-01-10 cs.CV cs.AI cs.CL cs.LG cs.MM 77%

OneLLM: One Framework to Align All Modalities with Language

Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, Xiangyu Yue

专题命中图文多模态：multimodal(abstract)；MLLM(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments Accepted by CVPR 2024. Code: https://github.com/csuhan/OneLLM

2412.18254 2024-12-25 cs.CV 77%

RaCMC: Residual-Aware Compensation Network with Multi-Granularity Constraints for Fake News Detection

Xinquan Yu, Ziqi Sheng, Wei Lu, Xiangyang Luo, Jiantao Zhou

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments 9 pages, 4 figures

2412.10758 2024-12-17 cs.CV 77%

Optimizing Vision-Language Interactions Through Decoder-Only Models

Kaito Tanaka, Benjamin Tan, Brian Wong

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

2411.16828 2024-11-27 cs.CV 77%

CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions

Yanqing Liu, Xianhang Li, Zeyu Wang, Bingchen Zhao, Cihang Xie

专题命中图文多模态：MLLM(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments 12 pages

2411.06764 2024-11-12 cs.CV cs.LG 77%

Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning

Hongsheng Zhang, Zhong Ji, Jingren Liu, Yanwei Pang, Jungong Han

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

2409.16183 2024-09-25 cs.CV 77%

Expert-level vision-language foundation model for real-world radiology and comprehensive evaluation

Xiaohong Liu, Guoxing Yang, Yulin Luo, Jiaji Mao, Xiang Zhang, Ming Gao, Shanghang Zhang, Jun Shen, Guangyu Wang

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

2306.11300 2024-09-10 cs.CV cs.AI cs.CL cs.MM 77%

RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing

Zilun Zhang, Tiancheng Zhao, Yulong Guo, Jianwei Yin

专题命中图文多模态：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments RS5M dataset v5

Journal ref TGRS-2023-06174

2407.20341 2024-07-31 cs.CV cs.AI cs.CL cs.MM 77%

BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues

Sara Sarto, Marcella Cornia, Lorenzo Baraldi, Rita Cucchiara

专题命中图文多模态：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments ECCV 2024

2406.12638 2024-06-19 cs.CV cs.LG 77%

Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model

Jiang-Xin Shi, Chi Zhang, Tong Wei, Yu-Feng Li

专题命中图文多模态：multi-modal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments Accepted by KDD 2024

2402.04252 2024-02-07 cs.CV 77%

EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters

Quan Sun, Jinsheng Wang, Qiying Yu, Yufeng Cui, Fan Zhang, Xiaosong Zhang, Xinlong Wang

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；multimodal foundation model(abstract)；分类 cs.CV

2401.16160 2024-01-31 cs.CV 77%

LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs

Shaoxiang Chen, Zequn Jie, Lin Ma

专题命中图文多模态：multimodal(abstract)；MLLM(abstract)；image-text(abstract)；分类 cs.CV

2312.14238 2024-01-18 cs.CV 77%

InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, Bin Li, Ping Luo, Tong Lu, Yu Qiao, Jifeng Dai

专题命中图文多模态：multimodal(abstract)；multi-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments 25 pages, 5 figures, 28 tables

2312.15011 2023-12-27 cs.CV 77%

Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases

Zhangyang Qi, Ye Fang, Mengchen Zhang, Zeyi Sun, Tong Wu, Ziwei Liu, Dahua Lin, Jiaqi Wang, Hengshuang Zhao

专题命中图文多模态：multimodal(abstract)；multi-modal(abstract)；multimodal foundation model(abstract)；分类 cs.CV

Comments Project Page: https://github.com/Qi-Zhangyang/Gemini-vs-GPT4V. arXiv admin note: substantial text overlap with arXiv:2309.17421

2306.07096 2023-12-07 cs.CV 77%

Global and Local Semantic Completion Learning for Vision-Language Pre-training

Rong-Cheng Tu, Yatai Ji, Jie Jiang, Weijie Kong, Chengfei Cai, Wenzhe Zhao, Hongfa Wang, Yujiu Yang, Wei Liu

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments arXiv admin note: substantial text overlap with arXiv:2211.13437

2310.10125 2023-10-17 cs.CV 77%

Few-shot Action Recognition with Captioning Foundation Models

Xiang Wang, Shiwei Zhang, Hangjie Yuan, Yingya Zhang, Changxin Gao, Deli Zhao, Nong Sang

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；multimodal foundation model(abstract)；分类 cs.CV

2305.11769 2023-08-08 cs.CV cs.AI cs.CL cs.MM 77%

Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner

Zikang Liu, Sihan Chen, Longteng Guo, Handong Li, Xingjian He, Jing Liu

专题命中图文多模态：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL、cs.AI

Comments 12 pages. Accepted by ACM MM '23

2307.14061 2023-07-27 cs.CV 77%

Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models

Dong Lu, Zhiqiang Wang, Teng Wang, Weili Guan, Hongchang Gao, Feng Zheng

专题命中图文多模态：multimodal(abstract)；cross-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments To appear in ICCV 2023

2306.00813 2023-06-02 cs.CV 77%

UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning

Xiao Dong, Runhui Huang, Xiaoyong Wei, Zequn Jie, Jianxing Yu, Jian Yin, Xiaodan Liang

专题命中图文多模态：multimodal(abstract)；multi-modal(abstract)；image-text(abstract)；分类 cs.CV

Comments NA

1. 图文多模态 4644 篇

Unexplored flaws in multiple-choice VQA evaluations

Audio-Enhanced Vision-Language Modeling with Latent Space Broadening for High Quality Data Expansion

AeroReformer: Aerial Referring Transformer for UAV-based Referring Image Segmentation

Describe, Adapt and Combine: Empowering CLIP Encoders for Open-set 3D Object Retrieval

Pushing the Limits of Safety: A Technical Report on the ATLAS Challenge 2025

Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs

MAKE: Multi-Aspect Knowledge-Enhanced Vision-Language Pretraining for Zero-shot Dermatological Assessment

Recent Advances in Out-of-Distribution Detection with CLIP-Like Models: A Survey

Can a Large Language Model Assess Urban Design Quality? Evaluating Walkability Metrics Across Expertise Levels

The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer

Hyperbolic Safety-Aware Vision-Language Models

ParGo: Bridging Vision-Language with Partial and Global Views

OneLLM: One Framework to Align All Modalities with Language

RaCMC: Residual-Aware Compensation Network with Multi-Granularity Constraints for Fake News Detection

Optimizing Vision-Language Interactions Through Decoder-Only Models

CLIPS: An Enhanced CLIP Framework for Learning with Synthetic Captions

Multi-Stage Knowledge Integration of Vision-Language Models for Continual Learning

Expert-level vision-language foundation model for real-world radiology and comprehensive evaluation

RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensing

BRIDGE: Bridging Gaps in Image Captioning Evaluation with Stronger Visual Cues

Efficient and Long-Tailed Generalization for Pre-trained Vision-Language Model

EVA-CLIP-18B: Scaling CLIP to 18 Billion Parameters

LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs

InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks

Gemini vs GPT-4V: A Preliminary Comparison and Combination of Vision-Language Models Through Qualitative Cases

Global and Local Semantic Completion Learning for Vision-Language Pre-training

Few-shot Action Recognition with Captioning Foundation Models

Enhancing Vision-Language Pre-Training with Jointly Learned Questioner and Dense Captioner

Set-level Guidance Attack: Boosting Adversarial Transferability of Vision-Language Pre-training Models

UniDiff: Advancing Vision-Language Models with Generative and Discriminative Learning