Retrieval-Based Brain Decoding by Alignment, not Complexity
基于对齐而非复杂性的检索式脑解码
Matteo Ciferri, Matteo Ferrante, Nicola Toschi
AI总结 本文通过跨多数据集实验证明,线性对比解码器在脑解码中优于岭回归和标准非线性方法,表明解码增益更多来自训练目标而非架构复杂性。
详情
认知科学中的一个著名理论认为,大脑中的概念被组织为高维向量,语义含义由该空间中的方向和相对角度捕获。脑解码是从神经活动中重建或检索刺激(或其表示)的努力,涉及找到一个近似大脑如何表示概念的函数。这激发了对对比目标作为逆转脑损失函数的生物合理候选者的研究。在这项工作中,我们研究了如何将功能磁共振成像(fMRI)活动与视觉、语言和音频基础模型的嵌入空间进行一般性映射。尽管神经计算在微观尺度上是高度非线性的,但fMRI测量平均了跨空间和时间的信号,并进一步被噪声平滑,从而有效地线性化了可观察的表示。与这些观点一致,我们在多个数据集上的实验表明,线性对比解码器始终优于岭回归和标准非线性替代方案,并且这些结果在图像、文本和声音中普遍适用。这些发现表明,解码增益更多地来自训练目标的选择而非架构复杂性,指向对比线性模型作为脑解码的原则性策略。
A prominent theory in cognitive science suggests that concepts in the brain are organized as high-dimensional vectors, with semantic meaning captured by directions and relative angles in this space. Brain decoding is the effort of reconstructing or retrieving stimuli (or their representations) from neural activity and involves finding a function that approximates how the brain represents concepts. This motivates the investigation of contrastive objectives as biologically plausible candidates to reverse the brain loss function. In this work, we study how functional MRI (fMRI) activity can generally be mapped with the embedding spaces of foundation models in vision, language, and audio. Although neural computations are highly non-linear at the microscale, fMRI measurements average signals across space and time, further smoothed by noise, effectively linearizing the observable representation. Consistent with these views, our experiments across multiple datasets demonstrate that linear contrastive decoders consistently outperform ridge regression and standard non-linear alternatives, and that these results generalize across images, text, and sound. These findings indicate that decoding gains arise more from the choice of training objective than from architectural complexity, pointing to contrastive-linear models as a principled strategy for brain decoding.