Discovering Functionally Selective Brain Regions with a Deep Topographic Multimodal Model
发现功能选择性脑区:一种深度地形多模态模型
Badr AlKhamissi, Johannes Mehrer, Lara Marinov, Ahmed Abdelaal, Abdulkadir Gokce, Martin Schrimpf
AI总结 提出Topo-Omni模型,通过空间平滑微调预训练基础模型,在单一连续虚拟皮层上整合视觉、听觉和语言/认知处理,产生与人类神经影像一致的多模态聚类,并用于发现新脑区。
详情
- Comments
- Preprint. First two author contributed equally
皮层中的邻近神经元具有相似的反应特征,从而在感觉和认知系统中产生系统性的空间组织。最近的地形模型再现了这种结构的某些方面,但仍然是单模态的,并且对每一层分别施加空间约束,产生了碎片化的图谱,既不能捕捉皮层处理流的连续性,也不能捕捉跨模态的整合。我们引入了Topo-Omni,一种地形多模态模型,其中视觉、听觉和语言/认知处理共享一个单一的连续虚拟皮层。通过使用空间平滑目标微调预训练的基础模型,该架构在跨模态中发展出与人类神经影像一致的聚类,从感觉系统到认知系统。驱动或抑制一个聚类会选择性偏向或损害感知,这与人类干预研究相似。最后,我们使用我们的模型在虚拟皮层中筛选新的聚类,并发现了新的自然景观和动物网络,并在人类数据中验证了它们。因此,单一的空间原则组织了跨模态和处理阶段的表征,产生了关于皮层组织的可检验假设。
Nearby neurons in cortex share similar response profiles, producing systematic spatial organization across sensory and cognitive systems. Recent topographic models reproduce aspects of this structure but remain unimodal and spatially constrain each layer separately, yielding fragmented maps that capture neither the contiguity of cortical processing streams nor their integration across modalities. We introduce Topo-Omni, a topographic multimodal model in which visual, auditory, and language/cognitive processing share a single contiguous in-silico sheet. Built by fine-tuning a pretrained foundation model with a spatial smoothness objective, this architecture develops clusters across modalities that are consistent with human neuroimaging, from sensory to cognitive systems. Driving or suppressing a cluster selectively biases or impairs perception, paralleling human intervention studies. Finally, we use our model to screen for novel clusters in-silico and discover new natural landscape and animal networks which we validate in human data. A single spatial principle thus organizes representations across modalities and processing stages, yielding testable hypotheses about cortical organization.