🤖 AI Summary
This study addresses the challenge of limited training data in electrocorticography (ECoG)-based neural encoding models, which stems from the scarcity of implantable patients. To overcome this limitation, the authors propose fine-tuning language representation models using non-invasive functional magnetic resonance imaging (fMRI) data and transferring the learned representations to model high spatiotemporal resolution ECoG signals. The work demonstrates for the first time that fMRI data—despite its temporal resolution being two orders of magnitude lower than ECoG—can significantly enhance ECoG prediction performance, thereby validating the efficacy of cross-modal transfer and data augmentation. Experimental results show that the fine-tuned models yield substantially improved ECoG predictions, with performance consistently increasing as more fMRI data are incorporated, and maintain robust generalization even under temporally downsampled conditions.
📝 Abstract
Neuroscientists have recently turned to intracranial brain recording methods, like electrocorticography (ECoG), for human experiments because of the fine spatial and temporal resolution that they afford. Models trained on this data, however, are fundamentally restricted by the patient populations that can receive the implants necessary for recording. We propose using non-invasive fMRI to bridge the gap in training data. Using spoken language representations fine-tuned on fMRI, we build encoding models of ECoG. These representations showed improved prediction performance in ECoG, even though the temporal resolution of fMRI is two orders of magnitude worse. Prediction improved in frequency bands well beyond what is directly measured in fMRI. Next, to test the procedure's generalization ability, we fine-tuned models on fMRI responses that were temporally downsampled by a factor of 2. Despite the loss in resolution, these models were able to predict fMRI and ECoG responses at levels comparable to the original fMRI-tuned models. Finally, we showed that ECoG performance steadily scales with the amount of fMRI-tuning data. Our results show that "slow" data like fMRI can be a valuable resource for building better models of "fast" brain data like ECoG. In the future, integrating across multiple recording methods may further improve performance in other applications, like decoding.