Decoding the Multimodal Mind: Generalizable Brain-to-Text Translation via Multimodal Alignment and Adaptive Routing
解码多模态思维:通过多模态对齐和自适应路由实现可泛化的脑到文本翻译
机构 * State Key Laboratory of Multimodal Artificial Intelligence System, Institute of Automation, Chinese Academy of Sciences(多模态人工智能系统国家重点实验室,中国科学院自动化研究所) ; School of Artificial Intelligence, University of Chinese Academy of Sciences(中国科学院大学人工智能学院) ; Department of Computer Science, The University of Manchester(曼彻斯特大学计算机科学系) ; Department of Language Science and Technology, Hong Kong Polytechnic University(香港理工大学语言科学与技术系)
专题命中 EEG解码 :EEG(summary_cn,abstract);BCI(abstract,abstract_cn);brain-computer interface(abstract)
AI总结 该研究针对脑机接口从人脑解码语言的挑战,提出利用多模态大语言模型和路由模块的统一框架,通过多模态对齐和自适应路由将脑信号与多模态语义空间对齐,在fMRI等数据集实验中性能领先,还扩展到EEG和MEG数据,为现实应用提供灵活方案。
Comments Accepted to ACL 2026 Findings