视觉大模型 / VLM

EVE: Efficient Vision-Language Pre-training with Masked Prediction and Modality-Aware MoE

Junyi Chen, Longteng Guo, Jia Sun, Shuai Shao, Zehuan Yuan, Liang Lin, Dongyu Zhang

专题命中 视觉问答 ：vision-language model(abstract)；visual reasoning(abstract)；visual question answering(abstract)；分类 cs.CV、cs.LG

Comments Accepted by AAAI 2024

PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs

Soroush Nasiriany, Fei Xia, Wenhao Yu, Ted Xiao, Jacky Liang, Ishita Dasgupta, Annie Xie, Danny Driess, Ayzaan Wahid, Zhuo Xu, Quan Vuong, Tingnan Zhang, Tsang-Wei Edward Lee, Kuang-Huei Lee, Peng Xu, Sean Kirmani, Yuke Zhu, Andy Zeng, Karol Hausman, Nicolas Heess, Chelsea Finn, Sergey Levine, Brian Ichter

专题命中 视觉问答 ：vision language model(abstract)；VLM(abstract)；visual question answering(abstract)；分类 cs.CV、cs.LG

MAMO: Masked Multimodal Modeling for Fine-Grained Vision-Language Representation Learning

Zijia Zhao, Longteng Guo, Xingjian He, Shuai Shao, Zehuan Yuan, Jing Liu

专题命中 视觉问答 ：visual reasoning(abstract)；visual question answering(abstract)；grounding(abstract)；分类 cs.CV、cs.LG

Comments SIGIR 2023, 10 pages

LitTraceQA: A Benchmark for Multi-Stage Grounding and Verification in Scientific Question Answering

LitTraceQA：面向科学问答的多阶段溯源与验证基准

Xuye Liu, Yimu Wang, Peng Shi, Bo Xue, Xiangrui Ke, Songcheng Cai, Kath Choi, Di Wu, Freda Shi, Krzysztof Czarnecki

机构 * University of Waterloo（滑铁卢大学） ； Amazon（亚马逊公司） ； City University of Hong Kong（香港城市大学） ； University of Amsterdam（阿姆斯特丹大学） 纠错

专题命中 视觉问答 ：grounding(title,abstract)

AI总结 该研究提出LitTraceQA基准，用于科学问答的多阶段溯源与验证，提供含多类型证据的问答数据，可评估系统的论文检索、证据溯源及答案准确性，助力生成可验证的科学问答结果。

Comments Work in Progress

SpaCE: Rethinking Spatial Capacity and Generalization in Multi-Frame Multimodal Large Language Models

SpaCE: 重新思考多帧多模态大语言模型中的空间容量与泛化能力

Mariana Costa, Camila Ferreira, Alberlucia Rafael Soarez, Alejandro Torres

专题命中 视觉问答 ：multimodal large language model(title)；visual question answering(abstract)

AI总结 提出SpaCE理论框架，证明多帧空间推理的信息论上界、样本复杂度界、PAC-Bayes泛化界，并刻画显式3D表示与隐式推理的偏差-方差权衡，在多个基准上验证了理论紧致性。

Knowledge-Graph Grounding Helps LLMs Only for Out-of-Training Knowledge: A Controlled Study on Clinical Question Answering

知识图谱基础仅对训练外知识帮助LLM：临床问答的受控研究

Madhulatha Mandarapu, Sandeep Kunkunuru

专题命中 视觉问答 ：grounding(title,abstract)

AI总结 通过受控实验发现，知识图谱基础仅在问题涉及模型训练外知识时提升LLM性能，对已知事实无帮助，且公共KG信息冗余。

Comments 9 pages. Code: https://github.com/samyama-ai/clinical-llm-graphrag

Calibrated Triage, Not Autonomy: Confidence Estimation for Medical Vision-Language Models

校准的分诊，而非自主：医学视觉-语言模型的置信度估计

Reza Khanmohammadi, Kundan Thind, Mohammad M. Ghassemi

机构 * Michigan State University（密歇根州立大学） 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)

AI总结 针对医学视觉-语言模型在回答时可能忽略图像而依赖语言先验的问题，提出使用置信度估计进行校准分诊，通过评估七种置信度估计器，发现高置信度区域是区分可用估计器的关键，最佳探针可将错误率从41-45%降至1-4%，但无估计器在所有领域和模型中一致最优。

Revisiting Greedy Decoding for Visual Question Answering: A Calibration Perspective

重新审视视觉问答中的贪婪解码：一种校准视角

Boqi Chen, Xudong Liu, Yunke Ao, Jianing Qiu

机构 * ETH Zurich（苏黎世联邦理工学院） ； University of Toronto（多伦多大学） ； MBZUAI（穆桑比克人工智能研究所） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)

AI总结 针对视觉问答任务，从校准角度理论证明贪婪解码优于随机采样，并提出适用于推理模型的贪婪解码方法，实验验证其有效性。

Lifelong Learning in Vision-Language Models: Enhanced EWC with Cross-Modal Knowledge Retention

视觉-语言模型的终身学习：增强的EWC与跨模态知识保留

Hamza Ahmed Durrani, Rafay Suleman Durrani

机构 * Sejong University, Computer Science Engineering（世宗大学，计算机科学工程） ； Technische Universität Ilmenau, Computer Engineering（伊门瑙技术大学，计算机工程） 纠错

专题命中 视觉问答 ：vision-language model(title)；visual question answering(abstract)

AI总结 本文提出一种结合增强EWC与参数高效微调的框架，有效减少视觉-语言模型在连续学习中的遗忘，通过多模态Fisher信息矩阵计算和跨模态一致性保留，实现78%的遗忘率降低，并在动态环境中提升多模态AI系统的适应能力。

Comments 8 pages, 5 figures, 1 table. Applications in autonomous driving, intelligent robotic assistants, and adaptive robotics systems

Doc-V*:Coarse-to-Fine Interactive Visual Reasoning for Multi-Page Document VQA

Doc-V*：多页文档视觉问答中的粗到细交互推理

Yuanlei Zheng, Pei Fu, Hang Li, Ziyang Wang, Yuyi Zhang, Wenyu Ruan, Xiaojin Zhang, Zhongyu Wei, Zhenbo Luo, Jian Luan, Wei Chen, Xiang Bai

机构 * School of Software Engineering, Huazhong University of Science and Technology（华中科技大学软件工程学院） ； MiLM Plus, Xiaomi Inc.（小米公司MiLM Plus团队） ； School of Computer Science and Technology, Huazhong University of Science and Technology（华中科技大学计算机科学与技术学院） ； School of Data Science, Fudan University（复旦大学数据科学学院） 纠错

专题命中 视觉问答 ：visual reasoning(title)；visual question answering(abstract)

AI总结 Doc-V*提出一种无OCR的智能框架，通过序列证据聚合解决多页文档VQA问题，结合语义检索和目标页面获取，提升回答准确性和证据收集效率，在五个基准测试中优于开源基线，提升领域外性能达47.9%。

MM-Doc-R1: Training Agents for Long Document Visual Question Answering through Multi-turn Reinforcement Learning

MM-Doc-R1: 通过多轮强化学习训练代理进行长文档视觉问答

Jiahang Lin, Kai Hu, Binghai Wang, Yuhao Zhou, Zhiheng Xi, Honglin Guo, Shichun Liu, Junzhe Wang, Shihan Dou, Enyu Zhou, Hang Yan, Zhenhua Han, Tao Gui, Qi Zhang, Xuanjing Huang

机构 * Fudan University（复旦大学） ； Shanghai Qiji Zhifeng Co., Ltd（上海启智锋科技有限公司） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)

AI总结 MM-Doc-R1通过多轮强化学习训练代理，利用相似性基于策略优化提升长文档视觉问答性能，实验表明其优于现有方法。

JW-VL: A Vision-Language Model for Solar Physics

JW-VL：用于太阳物理的视觉-语言模型

Mingfu Shao, Hui Wang, Liyue Tong, Yuyang Li, Cunshi Wang, Jiaben Lin, Suo Liu, Haiqing Xu, Yin Zhang, Jing Huang

专题命中 视觉问答 ：vision-language model(title,abstract)

AI总结 本文提出JW-VL模型，专为太阳物理设计，整合多波段观测数据，实现从原始观测数据到图像识别、活动分析和OCR的端到端处理，同时构建多波段图像基准数据集。

Comments 16 Pages,8figures

ViTextVQA: A Large-Scale Visual Question Answering Dataset and a Novel Multimodal Feature Fusion Method for Vietnamese Text Comprehension in Images

ViTextVQA: 一个大规模视觉问答数据集和一种新的多模态特征融合方法用于图像中的越南语文本理解

Quan Van Nguyen, Dan Quang Tran, Huy Quang Pham, Thang Kien-Bao Nguyen, Nghia Hieu Nguyen, Kiet Van Nguyen, Ngan Luu-Thuy Nguyen

机构 * Faculty of Information Science and Engineering, University of Information Technology（信息科学与工程学院，信息科技大学） ； Vietnam National University（越南国家大学） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)

AI总结 ViTextVQA是一个大规模视觉问答数据集，提出了一种新的多模态特征融合方法，用于提升图像中越南语文本的理解能力。

Comments International Journal of Expert Systems with Applications

Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models

视觉语言模型中实体知识提取跨模态性能差距

Ido Cohen, Daniela Gottesman, Mor Geva, Raja Giryes

机构 * Tel Aviv University（特拉维夫大学） 纠错

专题命中 视觉问答 ：vision language model(title)；vision-language model(abstract)

AI总结 研究视觉语言模型在跨模态实体知识提取中的性能差异，揭示信息流动限制及模型推理效率问题。

Comments Accepted to ACL 2025 Main Conference

LAVQA: A Latency-Aware Visual Question Answering Framework for Shared Autonomy in Self-Driving Vehicles

Shuangyu Xie, Kaiyuan Chen, Wenjing Chen, Chengyuan Qian, Christian Juette, Liu Ren, Dezhen Song, Ken Goldberg

机构 * University of California Berkeley（加州大学伯克利分校） ； Bosch Research（博世研究） ； MBZUAI ； Texas A&M University（德克萨斯农工大学） ； IEOR UC Berkeley（伯克利大学工业工程与运筹学系） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)

Distilling Multilingual Vision-Language Models: When Smaller Models Stay Multilingual

Sukrit Sriratanawilai, Jhayahgrit Thongwat, Romrawin Chumpu, Patomporn Payoungkhamdee, Sarana Nutanong, Peerat Limkonchotiwat

机构 * VISTEC ； AI Singapore 纠错

专题命中 视觉问答 ：vision-language model(title,abstract)

Comments Work in progress

AdaDocVQA: Adaptive Framework for Long Document Visual Question Answering in Low-Resource Settings

Haoxuan Li, Wei Song, Aofan Liu, Peiwu Qin

机构 * Tsinghua Shenzhen International Graduate School, Tsinghua University（清华大学深圳国际研究生院） ； School of Automation, Guangdong University of Technology（广东工业大学自动化学院） ； School of Information Engineering, Peking University（北京大学信息工程学院） 纠错

专题命中 视觉问答 ：visual question answering(title,abstract)

LiGT: Layout-infused Generative Transformer for Visual Question Answering on Vietnamese Receipts

Thanh-Phong Le, Trung Le Chi Phan, Nghia Hieu Nguyen, Kiet Van Nguyen

专题命中 视觉问答 ：visual question answering(title,abstract)

Comments Accepted at IJDAR

Application of Multimodal Large Language Models in Autonomous Driving

Md Robiul Islam

专题命中 视觉问答 ：multimodal large language model(title)；MLLM(abstract)

Comments 9 pages, 9 figures

FaultGPT: Industrial Fault Diagnosis Question Answering System by Vision Language Models

Jiao Chen, Ruyi Huang, Zuohong Lv, Jianhua Tang, Weihua Li

专题命中 视觉问答 ：vision language model(title)；vision-language model(abstract)

Towards Cross-Lingual Explanation of Artwork in Large-scale Vision Language Models

Shintaro Ozaki, Kazuki Hayashi, Yusuke Sakai, Hidetaka Kamigaito, Katsuhiko Hayashi, Taro Watanabe

专题命中 视觉问答 ：vision language model(title,abstract)

Comments NAACL 2025 (Findings)

Enhancing Financial VQA in Vision Language Models using Intermediate Structured Representations

Archita Srivastava, Abhas Kumar, Rajesh Kumar, Prabhakar Srinivasan

专题命中 视觉问答 ：vision language model(title)；分类 cs.CV、cs.AI、cs.LG

Discrete Subgraph Sampling for Interpretable Graph based Visual Question Answering

Pascal Tilli, Ngoc Thang Vu

专题命中 视觉问答 ：visual question answering(title,abstract)

Comments Accepted at COLING 2025

LIVE: Learnable In-Context Vector for Visual Question Answering

Yingzhe Peng, Chenduo Hao, Xu Yang, Jiawei Peng, Xinting Hu, Xin Geng

专题命中 视觉问答 ：visual question answering(title,abstract)

Multimodal Large Language Models and Tunings: Vision, Language, Sensors, Audio, and Beyond

Soyeon Caren Han, Feiqi Cao, Josiah Poon, Roberto Navigli

专题命中 视觉问答 ：multimodal large language model(title)；visual question answering(abstract)

Comments Accepted at ACM-MM 2024

A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answering

Yunxin Li, Longyue Wang, Baotian Hu, Xinyu Chen, Wanqi Zhong, Chenyang Lyu, Wei Wang, Min Zhang

专题命中 视觉问答 ：visual question answering(title,abstract)

Comments 20 pages, 15 pages; technical paper

BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining

Minjun Kim, Seungwoo Song, Youhan Lee, Haneol Jang, Kyungtae Lim

专题命中 视觉问答 ：visual question answering(title,abstract)

Precision Empowers, Excess Distracts: Visual Question Answering With Dynamically Infused Knowledge In Language Models

Manas Jhalani, Annervaz K M, Pushpak Bhattacharyya

专题命中 视觉问答 ：visual question answering(title,abstract)

Comments 16 pages, 12 figures

Translation Deserves Better: Analyzing Translation Artifacts in Cross-lingual Visual Question Answering

ChaeHun Park, Koanho Lee, Hyesu Lim, Jaeseok Kim, Junmo Park, Yu-Jung Heo, Du-Seong Chang, Jaegul Choo

专题命中 视觉问答 ：visual question answering(title,abstract)

Comments ACL 2024 Findings Accepted

Federated Document Visual Question Answering: A Pilot Study

Khanh Nguyen, Dimosthenis Karatzas

专题命中 视觉问答 ：visual question answering(title)；分类 cs.CV、cs.AI、cs.LG