多模态大模型

SmolDocling: An ultra-compact vision-language model for end-to-end multi-modal document conversion

Ahmed Nassar, Andres Marafioti, Matteo Omenetti, Maksym Lysak, Nikolaos Livathinos, Christoph Auer, Lucas Morin, Rafael Teixeira de Lima, Yusik Kim, A. Said Gurbuz, Michele Dolfi, Miquel Farré, Peter W. J. Staar

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Comments 24 pages, 10 figures

GCS-M3VLT: Guided Context Self-Attention based Multi-modal Medical Vision Language Transformer for Retinal Image Captioning

Teja Krishna Cherukuri, Nagur Shareef Shaik, Jyostna Devi Bodapati, Dong Hye Ye

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Comments This paper has been accepted for presentation at the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP 2025)

Attacks on multimodal models

Viacheslav Iablochnikov, Alexander Rogachev

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

Comments 19 pages, 13 figures, 3 tables

Multi-Modal CLIP-Informed Protein Editing

Mingze Yin, Hanjing Zhou, Yiheng Zhu, Miao Lin, Yixuan Wu, Jialu Wu, Hongxia Xu, Chang-Yu Hsieh, Tingjun Hou, Jintai Chen, Jian Wu

专题命中 图文多模态 ：multi-modal(title)；分类 cs.AI

Comments 13 pages, 7 figures, 5 tables

Journal ref Health Data Science, 2024

Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotations

Chenyu You, Yifei Min, Weicheng Dai, Jasjeet S. Sekhon, Lawrence Staib, James S. Duncan

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Comments Accepted by CVPR 2024

Towards Multimodal In-Context Learning for Vision & Language Models

Sivan Doveh, Shaked Perek, M. Jehanzeb Mirza, Wei Lin, Amit Alfassy, Assaf Arbelle, Shimon Ullman, Leonid Karlinsky

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

Enhancing Weakly Supervised Semantic Segmentation with Multi-modal Foundation Models: An End-to-End Approach

Elham Ravanbakhsh, Cheng Niu, Yongqing Liang, J. Ramanujam, Xin Li

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Fewshot learning on global multimodal embeddings for earth observation tasks

Matt Allen, Francisco Dorr, Joseph A. Gallego-Mejia, Laura Martínez-Ferrer, Anna Jungbluth, Freddie Kalaitzis, Raúl Ramos-Pollán

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

Comments 9 pages, 6 figures, presented on NeurIPS workshop on Robustness of Few-shot and Zero-shot Learning in Foundation Models

Multimodal Search on Iconclass using Vision-Language Pre-Trained Models

Cristian Santini, Etienne Posthumus, Mary Ann Tan, Oleksandra Bruns, Tabea Tietz, Harald Sack

专题命中 图文多模态 ：multimodal(title)；分类 cs.AI

Uncurated Image-Text Datasets: Shedding Light on Demographic Bias

Noa Garcia, Yusuke Hirota, Yankun Wu, Yuta Nakashima

专题命中 图文多模态 ：image-text(title)；分类 cs.CV

Comments CVPR 2023

Task-Oriented Multi-Modal Mutual Leaning for Vision-Language Models

Sifan Long, Zhen Zhao, Junkun Yuan, Zichang Tan, Jiangjiang Liu, Luping Zhou, Shengsheng Wang, Jingdong Wang

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

CLIP-Mesh: Generating textured meshes from text using pretrained image-text models

Nasir Mohammad Khalid, Tianhao Xie, Eugene Belilovsky, Tiberiu Popa

专题命中 图文多模态 ：image-text(title)；分类 cs.CV

Comments 8 pages, 8 figures, Accepted at SIGGRAPH ASIA 2022, Project Page at https://www.nasir.lol/clipmesh

How to find a good image-text embedding for remote sensing visual question answering?

Christel Chappuis, Sylvain Lobry, Benjamin Kellenberger, Bertrand Le Saux, Devis Tuia

专题命中 图文多模态 ：image-text(title)；分类 cs.CV

Comments 10 pages, 4 figures, presented in the MACLEAN workshop during ECML PKDD 2021

Multi-Modal Image Captioning for the Visually Impaired

Hiba Ahsan, Nikita Bhalla, Daivat Bhatt, Kaivankumar Shah

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CL

Comments 8 pages, 2 figures, 2 tables, accepted to NAACL-HLT SRW 2021

Multi-modal Feature Fusion with Feature Attention for VATEX Captioning Challenge 2020

Ke Lin, Zhuoxin Gan, Liwei Wang

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching

Chunxiao Liu, Zhendong Mao, An-An Liu, Tianzhu Zhang, Bin Wang, Yongdong Zhang

专题命中 图文多模态 ：image-text(title)；分类 cs.MM

Comments Accepted by ACMMM2019

Towards Unsupervised Image Captioning with Shared Multimodal Embeddings

Iro Laina, Christian Rupprecht, Nassir Navab

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

Comments ICCV 2019

Image Captioning with Clause-Focused Metrics in a Multi-Modal Setting for Marketing

Philipp Harzig, Dan Zecha, Rainer Lienhart, Carolin Kaiser, René Schallner

专题命中 图文多模态 ：multi-modal(title)；分类 cs.CV

Comments 6 pages, accepted at MIPR 2019

Multimodal Image Captioning for Marketing Analysis

Philipp Harzig, Stephan Brehm, Rainer Lienhart, Carolin Kaiser, René Schallner

专题命中 图文多模态 ：multimodal(title)；分类 cs.CV

Comments 4 pages, 1 figure, accepted at MIPR2018

Joint Image-Text News Topic Detection and Tracking with And-Or Graph Representation

Weixin Li, Jungseock Joo, Hang Qi, Song-Chun Zhu

专题命中 图文多模态 ：image-text(title)；分类 cs.CL

Look Clearly Before Answering: Mitigating Hallucinations in LVLMs via Saliency-Driven Perceptual Realignment

回答前看清楚：通过显著性驱动的感知重新对齐减轻LVLMs中的幻觉

Pengxu Chen, Yao Zhu, Guangming Zhu, Jun Sheng, Jincai Huang, Xiangyang Ji, Liang Zhang

机构 * Xidian University（西安电子科技大学） ； Tsinghua University（清华大学） ； Shanghai Road Transport Development Center（上海市道路运输发展中心） ； Hunan Institute of Advanced Technology（湖南先进技术研究院） 纠错

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.MM

AI总结 研究针对LVLMs易产生幻觉问题，提出无需训练的SDPR框架，通过显著性驱动注意力重新分配、缓存对齐及先验约束对比解码，整体对齐视觉意识，在多基准测试中优于现有方法，无需额外训练且开销小。

Comments Accepted by ACM Multimedia 2026

Prompt Engineering in Segment Anything Model: Methodologies, Applications, and Emerging Challenges

分割一切模型中的提示工程：方法、应用与新兴挑战

Yidong Jiang, Jiangtong Li, Daiwei Cheng

机构 * School of Computer Science and Technology, Tongji University（计算机科学与技术学院，同济大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

AI总结 本综述系统梳理了SAM的提示工程技术，将其分为三类并分析方法演变，探讨其跨领域应用，指出三大挑战并提出有前景的研究方向，为相关领域提供参考。

Intern-S2-Preview: Scientific Agentic Foundation Model

Intern-S2-Preview：科学智能体基础模型

Lei Bai, Jiaqi Cao, Chiyu Chen, Guanzhou Chen, Kai Chen, Guangran Cheng, Erfei Cui, Xuanlang Dai, Shengyuan Ding, Shangheng Du, Yanhui Duan, Yue Fan, Youqing Fang, Quan Gan, Yuanyuan Gao, Jiaye Ge, Lixin Gu, Yuzhe Gu, Qipeng Guo, Junjun He, Xin Hong, Ming Hu, Zhouqi Hua, Haian Huang, Junhao Huang, Zixian Huang, Minxi Jin, Lingkai Kong, Alexander Lam, Zehao Li, Zonglin Li, Tianhao Liang, Dahua Lin, Junyao Lin, Tianyang Lin, Zhouhan Lin, Jiangning Liu, Jin Liu, Kuikun Liu, Wenran Liu, Yifei Liu, Yuhong Liu, Yuhong Liu, Zhoumianze Liu, Ziyan Liu, Ziyu Liu, Haijun Lv, Han Lv, Chengqi Lyu, Le Ma, Ningsheng Ma, Zerun Ma, Haoyang Peng, Runyu Peng, Jifei Shan, Zixin Shang, Kou Shi, Xiang Shi, Qisheng Su, Xuerui Su, Hao Sun, Xiao Sun, Yanan Sun, Yu Sun, Huanze Tang, Yinghao Tang, Wenhui Tian, Zhongbo Tian, Bingli Wang, Haomin Wang, Jiarui Wang, Jingzhi Wang, Rui Wang, Xiquan Wang, Yi Wang, Zhecan Wang, Ziyi Wang, Zun Wang, Rubin Wei, Lianyi Wu, Wen Wu, Yue Wu, Yuhan Wu, Zhenyu Wu, Zijian Wu, Shuhao Xing, Jun Xu, Xingle Xu, Xuenan Xu, Xiangchao Yan, Ziang Yan, Bowen Yang, Danni Yang, Lin Yang, Zhiqi Yang, Qian Yao, Haochen Ye, Peng Ye, Jinhui Yin, Jiashuo Yu, Dingbo Yuan, Fei Yuan, Yuhang Zang, Bo Zhang, Chao Zhang, Chen Zhang, Hongjie Zhang, Junming Zhang, Wenlong Zhang, Wenwei Zhang, Yiming Zhang, Zhuo Zhang, Ziyang Zhang, Haiteng Zhao, Penghao Zhao, Yibo Zhao, Zhonghan Zhao, Zhihang Zhong, Bowen Zhou, Peiheng Zhou, Xin Zhou, Xinyu Zhou, Yunhua Zhou, Dongsheng Zhu, Yicheng Zou

机构 * Shanghai AI Laboratory（上海人工智能实验室） 纠错

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.CL

AI总结 本研究提出Intern-S2-Preview系列科学智能体基础模型，通过多阶段训练与架构优化，在多类基准上取得领先结果，相关模块可提升科学任务表现且无需修改主干模型。

Comments 35 pages, 12 figures

SpikeWorld: Fast-State Adaptation for Frozen Spiking World Models

SpikeWorld：用于冻结脉冲世界模型的快速状态适应

Ziqiao Yu

机构 * DiDi International Business Group（滴滴国际业务集团） 纠错

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 SpikeWorld是145万参数的稀疏脉冲模型，冻结训练参数后通过外部路径实现快速状态适应，提升多模态预测等性能，在Meta-World任务中提高冻结策略奖励与成功率。

Comments 14 pages, 2 figures, 4 tables. Code: https://github.com/Oooorca/SpikeWorld

Gaze Behavior in Visual World Experiments Can be Modeled With Off-the-shelf Language-Vision Encoders

视觉世界实验中的注视行为可采用现成的语言-视觉编码器建模

Rahul Murali Shankar, Titus von der Malsburg, Sebastian Padó

机构 * University of Stuttgart（斯图加特大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；multi-modal(abstract)；分类 cs.CV、cs.CL

AI总结 该研究提出结合CLIP双编码器与双模态归因方法的新方法，无需微调或生成式架构，即可复现经典视觉世界研究的人类注视行为预测结果。

Intersectional Fairness in Vision-Language Models for Medical Image Disease Classification

视觉-语言模型在医学影像疾病分类中的交叉公平性

Yupeng Zhang, Adam G. Dunn, Usman Naseem, Jinman Kim

机构 * Biomedical Data Analysis and Visualisation (BDAV) Lab, School of Computer Science（生物医学数据分析与可视化实验室，计算机科学学院） ； Sydney School of Public Health（悉尼公共卫生学院） ； School of Computing（计算学院） 纠错

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

AI总结 本文提出CMAC-MMD框架，通过标准化交叉亚组的诊断确定性，减少医学影像疾病分类中的交叉偏见，提升诊断准确性和公平性。

QR-Structured Thermal Triggers for Targeted Semantic Attacks on Infrared Vision-Language Models

用于红外视觉语言模型定向语义攻击的QR结构化热触发装置

Xiang Chen, Yingying Zhao, Chao Li, Jiaju Han, Ben Zhang, Ang Li, Jiahuan Long, Yiwei Wei, Jiujiang Guo, Chengyin Hu

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 本研究提出QR-STT框架，作为隐蔽黑盒攻击手段，可定向操控IR-VLMs的语义对齐，其扰动还能跨任务迁移，凸显了对该类攻击进行鲁棒性评估的必要性。

TextSLIP: Text Self-Supervised CLIP for Medical Report Generation

TextSLIP：用于医学报告生成的文本自监督CLIP

Haoyu Jiang, Ziping Cong

专题命中 图文多模态 ：cross-modal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 研究针对现有CLIP方法在医学报告生成中语义监督不足问题，提出TextSLIP框架，通过模态内文本对比学习增强CLIP，经实验验证其在报告生成指标上有改进，表明文本级对比学习对改善医学视觉-文本对齐有潜力。

Comments 9 pages

When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalities

当结构化稀疏自编码器跨模态学习一致概念时

Weiduo Liao, Yunqiao Yang, Ying Wei

机构 * Zhejiang University（浙江大学） ； Nanyang Technological University（南洋理工大学） 纠错

专题命中 图文多模态 ：multimodal(abstract)；cross-modal(abstract)；分类 cs.CV、cs.AI

AI总结 研究视觉语言模型中普通稀疏自编码器难以学习模态一致概念的问题，提出结构化稀疏自编码器$S^2AE$，通过图像块分组及结构化稀疏正则化强化概念一致性，经实验验证该方法在语义对齐等方面有提升，能促进跨模态表示更连贯解缠。

Geometry-Aware Uncertainty Coresets for Robust Visual In-Context Learning in Histopathology

面向几何的不确定性聚类用于病理学中鲁棒的视觉上下文学习

Franciskus Xaverius Erick, Johanna Paula Müller, Bernhard Kainz

机构 * FAU Erlangen-Nürnberg, Erlangen, DE（埃尔兰根-纽伦堡大学） ； Department of Computing, Imperial College London, London, UK（伦敦帝国理工学院计算机系） 纠错

专题命中 图文多模态 ：multimodal(abstract)；image-text(abstract)；分类 cs.CV、cs.AI

AI总结 本文提出GAUC，一种无需训练的聚类选择方法，直接在预训练的多模态嵌入空间中操作，通过优化三个目标提升视觉上下文学习的鲁棒性、准确性和校准性。