医学 AI

Brain-Adapter: Enhancing Neurological Disorder Analysis with Adapter-Tuning Multimodal Large Language Models

Jing Zhang, Xiaowei Yu, Yanjun Lyu, Lu Zhang, Tong Chen, Chao Cao, Yan Zhuang, Minheng Chen, Tianming Liu, Dajiang Zhu

专题命中 医疗多模态 ：medical image(abstract)；diagnosis(abstract)；分类 cs.CV、eess.IV

Exploring Compositional Generalization of Multimodal LLMs for Medical Imaging

Zhenyang Cai, Junying Chen, Rongsheng Wang, Weihong Wang, Yonglin Deng, Dingjie Song, Yize Chen, Zixu Zhang, Benyou Wang

机构 * The Chinese University of Hong Kong, Shenzhen（香港中文大学（深圳）） 纠错

专题命中 医疗多模态 ：medical image(abstract)；diagnosis(abstract)；分类 cs.CV、cs.LG

Progressive Local Alignment for Medical Multimodal Pre-training

Huimin Yan, Xian Yang, Liang Bai, Jiye Liang

专题命中 医疗多模态 ：medical image(abstract)；diagnosis(abstract)；分类 cs.CV、cs.LG

Comments We are currently revising the methodology described in the manuscript to improve its clarity. We have decided to withdraw the current version until a more robust and complete version is ready

Towards Robust Multimodal Representation: A Unified Approach with Adaptive Experts and Alignment

Nazanin Moradinasab, Saurav Sengupta, Jiebei Liu, Sana Syed, Donald E. Brown

专题命中 医疗多模态 ：medical image(abstract)；diagnosis(abstract)；分类 cs.CV、cs.LG

G2D: From Global to Dense Radiography Representation Learning via Vision-Language Pre-training

Che Liu, Cheng Ouyang, Sibo Cheng, Anand Shah, Wenjia Bai, Rossella Arcucci

专题命中 医疗多模态 ：medical image(abstract)；radiology(abstract)；分类 cs.CV、cs.LG

Comments Accepted by NeurIPS2024

Visual Question Answering in Ophthalmology: A Progressive and Practical Perspective

Xiaolan Chen, Ruoyu Chen, Pusheng Xu, Weiyi Zhang, Xianwen Shang, Mingguang He, Danli Shi

专题命中 医疗多模态 ：medical image(abstract)；diagnosis(abstract)；分类 cs.CV、eess.IV

Dr-LLaVA: Visual Instruction Tuning with Symbolic Clinical Grounding

Shenghuan Sun, Alexander Schubert, Gregory M. Goldgof, Zhiqing Sun, Thomas Hartvigsen, Atul J. Butte, Ahmed Alaa

专题命中 医疗多模态 ：medical image(abstract)；pathology(abstract)；分类 cs.CV、cs.LG

Comments Code available at: https://github.com/AlaaLab/Dr-LLaVA

Utilizing Synthetic Data for Medical Vision-Language Pre-training: Bypassing the Need for Real Images

Che Liu, Anand Shah, Wenjia Bai, Rossella Arcucci

专题命中 医疗多模态 ：medical image(abstract)；radiology(abstract)；分类 cs.CV、cs.LG

Comments Accepted by CVPR 2024 Workshop Data Curation and Augmentation in Enhancing Medical Imaging Applications

Multimodal Machine Learning in Image-Based and Clinical Biomedicine: Survey and Prospects

Elisa Warner, Joonsang Lee, William Hsu, Tanveer Syeda-Mahmood, Charles Kahn, Olivier Gevaert, Arvind Rao

专题命中 医疗多模态 ：medical image(abstract)；biomedical(abstract)；分类 cs.CV、cs.LG

Dynamic Multimodal Information Bottleneck for Multimodality Classification

Yingying Fang, Shuang Wu, Sheng Zhang, Chaoyan Huang, Tieyong Zeng, Xiaodan Xing, Simon Walsh, Guang Yang

专题命中 医疗多模态 ：diagnosis(abstract)；biomedical(abstract)；分类 cs.CV、eess.IV

Comments WACV 2024

Multimodal Identification of Alzheimer's Disease: A Review

Guian Fang, Mengsha Liu, Yi Zhong, Zhuolin Zhang, Jiehui Huang, Zhenchao Tang, Calvin Yu-Chian Chen

专题命中 医疗多模态 ：MRI(abstract)；diagnosis(abstract)；分类 cs.CV、eess.IV

Diagnosing Alzheimer's Disease using Early-Late Multimodal Data Fusion with Jacobian Maps

Yasmine Mustafa, Tie Luo

专题命中 医疗多模态 ：MRI(abstract)；CT(abstract)；分类 cs.CV、cs.LG

Comments To be published in Proceedings of 2023 IEEE Healthcom, December 2023

UnICLAM:Contrastive Representation Learning with Adversarial Masking for Unified and Interpretable Medical Vision Question Answering

Chenlu Zhan, Peng Peng, Hongsen Wang, Tao Chen, Hongwei Wang

专题命中 医疗多模态 ：diagnosis(abstract)；radiology(abstract)；分类 cs.CV、cs.LG

Clinically-Inspired Multi-Agent Transformers for Disease Trajectory Forecasting from Multimodal Data

Huy Hoang Nguyen, Matthew B. Blaschko, Simo Saarakkala, Aleksei Tiulpin

专题命中 医疗多模态 ：medical image(abstract)；diagnosis(abstract)；分类 cs.CV、eess.IV

Comments Accepted in IEEE Transactions on Medical Imaging 2023

IIHT: Medical Report Generation with Image-to-Indicator Hierarchical Transformer

Keqiang Fan, Xiaohao Cai, Mahesan Niranjan

专题命中 医疗多模态 ：medical image(abstract)；diagnosis(abstract)；分类 cs.CV、cs.LG

Multimodal Brain Disease Classification with Functional Interaction Learning from Single fMRI Volume

Wei Dai, Ziyao Zhang, Lixia Tian, Shengyuan Yu, Shuhui Wang, Zhao Dong, Hairong Zheng

专题命中 医疗多模态 ：MRI(abstract)；diagnosis(abstract)；分类 cs.CV、eess.IV

Can representation learning for multimodal image registration be improved by supervision of intermediate layers?

Elisabeth Wetzer, Joakim Lindblad, Nataša Sladoje

专题命中 医疗多模态 ：medical image(abstract)；biomedical(abstract)；分类 cs.CV、cs.LG

Comments 15 Pages + 9 Pages Appendix, 10 Figures

Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domains

Pierre Chambon, Christian Bluethgen, Curtis P. Langlotz, Akshay Chaudhari

专题命中 医疗多模态 ：medical image(abstract)；radiology(abstract)；分类 cs.CV、cs.LG

Comments 17 pages, 8 figures

Journal ref Foundation Models for Decision Making Workshop at Neural Information Processing Systems, 2022

Is Image-to-Image Translation the Panacea for Multimodal Image Registration? A Comparative Study

Jiahao Lu, Johan Öfverstedt, Joakim Lindblad, Nataša Sladoje

专题命中 医疗多模态 ：medical image(abstract)；biomedical(abstract)；分类 cs.CV、eess.IV

Comments 37 pages, 10 figures, 2 tables

Groupwise Multimodal Image Registration using Joint Total Variation

Mikael Brudfors, Yaël Balbastre, John Ashburner

专题命中 医疗多模态 ：MRI(abstract)；CT(abstract)；分类 cs.CV、eess.IV

MMBERT: Multimodal BERT Pretraining for Improved Medical VQA

Yash Khare, Viraj Bagal, Minesh Mathew, Adithi Devi, U Deva Priyakumar, CV Jawahar

专题命中 医疗多模态 ：medical image(abstract)；radiology(abstract)；分类 cs.CV、cs.LG

Unsupervised Multimodal Image Registration with Adaptative Gradient Guidance

Zhe Xu, Jiangpeng Yan, Jie Luo, Xiu Li, Jayender Jagadeesan

专题命中 医疗多模态 ：MRI(abstract)；CT(abstract)；分类 cs.CV、eess.IV

Comments 5 pages, under review

Auxiliary Signal-Guided Knowledge Encoder-Decoder for Medical Report Generation

Mingjie Li, Fuyu Wang, Xiaojun Chang, Xiaodan Liang

专题命中 医疗多模态 ：medical image(abstract)；CT(abstract)；分类 cs.CV、eess.IV

Comments 11 pages, 3 figures

Loud or Silent? A Reusable Framework for Per-Modality Failure Analysis in Multimodal Clinical AI

大声还是沉默？一种用于多模态临床AI中模态级失败分析的可复用框架

Quang Bui, Shlok Jaiswal, Samuel Paik-Heintz, Kevin Zhou, Kaushik Madapati, Krittaphas Chaisutyakorn, Noah Dane Hebdon, Dimitrios Proios, Sebastián Andrés Cajas Ordóñez, Kacper Dobek, Boya Zhang, Aly Dhedhi, Ahram Han, Kushul Reddy Palakala, Rahul Gorijavolu, Jacques Kpodonu, Leo Anthony Celi

机构 * Massachusetts Institute of Technology（麻省理工学院） ； American International School Vienna（维也纳美国国际学校） ； Neuqua Valley High School（纽夸谷高中） ； North Hollywood High School（北好莱坞高中） ； Hopewell Valley Central High School（霍普韦尔谷中央高中） ； University of California, Berkeley（加州大学伯克利分校） ； Siriraj Hospital（诗里拉吉医院） ； Harvard University（哈佛大学） ； Agency for Science, Technology and Research（新加坡科技研究局） ； Johns Hopkins University（约翰斯·霍普金斯大学） ； University of Geneva（日内瓦大学） ； Poznan University of Technology（波兹南理工大学） 纠错

专题命中 医疗多模态 ：clinical AI(title)

AI总结 该研究提出一种模型无关的多模态临床AI模态级失败分析框架，可复用且仅用部署可观测信号，经植入真实值和MIMIC-IV队列验证，能定位失败模态、区分失败类型，为心脏基础模型部署提供参考。

RetiBridge: Bridging Quantitative Retinal Biomarkers and Qualitative Diagnosis with a Knowledge-Guided Multimodal Large Language Model

RetiBridge：用知识引导的多模态大语言模型连接定量视网膜生物标志物与定性诊断

Zhuangzhi Gao, Hongyi Qin, He Zhao, Qinkai Yu, Feixiang Zhou, Fu Wang, Jinru Ding, Eduard Shantsila, Uazman Alam, Alena Shantsila, Wahbi El-Bouri, Gregory Y. H. Lip, Yalin Zheng

机构 * University of Liverpool（利物浦大学） ； Institute of Life Course & Medical Sciences（生命课程与医学科学研究院） ； Department of Eye and Vision Sciences（眼科与视觉科学系） ； Computer Science Department（计算机科学系） ； Cardiovascular & Metabolic Medicine（心血管与代谢医学） ； Liverpool Centre for Cardiovascular Science（利物浦心血管科学中心） 纠错

专题命中 医疗多模态 ：diagnosis(title)

AI总结 该研究提出RetiBridge模型，结合CFP、OCT数据与文本，通过知识引导微调学习定量到定性诊断路径，在眼科理解基准上表现优于开源基线及OpenAI o3，代码数据已公开。

Comments 10 pages, 4 figures, 3 table. Equal contribution: Zhuangzhi Gao and Hongyi Qin. Corresponding author: Yalin Zheng (yzheng@liverpool.ac.uk)

Why Does Grounding Hurt Medical VQA? Benchmarking, Diagnosis, and Fine-Tuning of Vision-Language Models

对前沿视觉-语言模型进行审计以实现可信的医学视觉问答：定位失败、格式崩溃和领域适应

Xupeng Chen, Binbin Shi, Chenqian Le, Qifu Yin, Lang Lin, Haowei Ni, Ran Gong, Panfeng Li

机构 * New York University, New York, USA（纽约大学） ； Tsinghua University, Beijing, China（清华大学） ； Columbia University, New York, USA（哥伦比亚大学） ； University of Michigan, Ann Arbor, USA（密歇根大学） 纠错

专题命中 医疗多模态 ：diagnosis(title)

AI总结 本文审计了五种前沿视觉-语言模型在医学视觉问答中的表现，发现定位失败和格式崩溃是信任瓶颈，通过领域适应可提升性能。

Holmes: Multimodal Agentic Diagnosis for Mixed-Language Mobile Crashes at Industrial Scale

Holmes: 工业级混合语言移动崩溃的多模态智能诊断

Jia Li, Wenyuan Ma, Ting Peng, Haibin Zheng, Yuetang Deng

机构 * The Chinese University of Hong Kong（香港中文大学） ； Tencent Inc.（腾讯公司） 纠错

专题命中 医疗多模态 ：diagnosis(title)

AI总结 提出多智能体系统Holmes，通过多模态运行时信号合成重构崩溃上下文，采用层次化检索-探索-推理架构，在70M行代码中实现87.6%函数级定位准确率，将平均排查时间降低98%至约77秒。

Comments Accepted at FSE'26 Industry Track

MARCUS: An agentic, multimodal vision-language model for cardiac diagnosis and management

MARCUS：一种用于心脏诊断和管理的代理式多模态视觉-语言模型

Jack W O'Sullivan, Mohammad Asadi, Lennart Elbe, Akshay Chaudhari, Tahoura Nedaee, Francois Haddad, Michael Salerno, Li Fe-Fei, Ehsan Adeli, Rima Arnaout, Euan A Ashley

机构 * Division of Cardiology, Department of Medicine, Stanford University（斯坦福大学心脏病学系） ； Department of Biomedical Data Science, Stanford University（斯坦福大学生物医学数据科学系） ； Department of Medicine, Radiology, and Pediatrics, UCSF（旧金山大学医学系、放射学与儿科学系） ； Bakar Institute, UCSF（Bakar研究所，旧金山大学） ； UCSF–UC Berkeley Joint Program in Computational Precision Health（旧金山大学-伯克利计算精准健康联合计划） ； Department of Radiology, Stanford University（斯坦福大学放射学系） ； Department of Psychiatry and Behavioral Sciences, Stanford University（斯坦福大学精神病学与行为科学系） ； Department of Computer Science, Stanford University（斯坦福大学计算机科学系） ； Department of Electrical Engineering, Stanford University（斯坦福大学电气工程系） ； Department of Biology, Stanford University（斯坦福大学生物学系） 纠错

专题命中 医疗多模态 ：diagnosis(title)

AI总结 MARCUS通过多模态视觉-语言模型实现心电图、超声和心脏磁共振成像的端到端解读，其多代理架构在心脏诊断中表现出优于前沿模型的准确率和自由文本质量。

Generative vector search to improve pathology foundation models across multimodal vision-language tasks

生成向量搜索以提升多模态视觉-语言任务中的病理基础模型

Markus Ekvall, Ludvig Bergenstråhle, Patrick Truong, Ben Murrell, Joakim Lundeberg

专题命中 医疗多模态 ：pathology(title)

AI总结 STHLM通过生成向量搜索方法提升多模态视觉-语言任务中病理基础模型的检索性能，实现10-30%的性能提升和10倍的维度压缩

Comments 13 pages main (54 total), 2 main figures (9 total)

Rethinking Test-Time Scaling for Medical AI: Model and Task-Aware Strategies for LLMs and VLMs

Gyutaek Oh, Seoyeon Kim, Sangjoon Park, Byung-Hoon Kim

机构 * Department of Biomedical Systems Informatics, Yonsei University College of Medicine（生物医学系统信息学系，延世大学医学院） ； Yonsei University College of Medicine（延世大学医学院） ； Department of Radiation Oncology, Yonsei University College of Medicine（放射肿瘤学系，延世大学医学院） ； Department of Biomedical Systems Informatics and the Department of Psychiatry, Yonsei University College of Medicine（生物医学系统信息学系和精神病学系，延世大学医学院） ； Institute for Innovation in Digital Healthcare, Yonsei University（数字医疗创新研究所，延世大学） 纠错

专题命中 医疗多模态 ：medical AI(title)

Comments 11 pages, 6 figures