The Wisdom of a Crowd of Brains: A Universal Brain Encoder

2024-06-18

arXiv.org

AI Summary

Existing fMRI encoders are typically limited to single-subject settings and specific datasets, exhibiting poor generalizability. This work introduces the first universal brain encoder capable of cross-subject, cross-scanner (3T/7T), and cross-dataset fMRI response prediction. Our core innovation is a voxel-centric encoding architecture: it learns dedicated embeddings for each brain voxel and employs voxel–image cross-attention to automatically infer functional roles; further enhanced by multi-level image feature extraction and joint training on diverse fMRI sources. Experiments demonstrate substantial improvements in individual encoding accuracy, efficient transfer learning with minimal target-subject data, and generation of interpretable, voxel-wise functional maps. This framework advances neural mechanism interpretation by providing a robust, generalizable, and biologically grounded computational tool for brain–behavior mapping.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Computer Vision: Multi-modal Vision
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

Image-to-fMRI encoding is important for both neuroscience research and practical applications. However, such"Brain-Encoders"have been typically trained per-subject and per fMRI-dataset, thus restricted to very limited training data. In this paper we propose a Universal Brain-Encoder, which can be trained jointly on data from many different subjects/datasets/machines. What makes this possible is our new voxel-centric Encoder architecture, which learns a unique"voxel-embedding"per brain-voxel. Our Encoder trains to predict the response of each brain-voxel on every image, by directly computing the cross-attention between the brain-voxel embedding and multi-level deep image features. This voxel-centric architecture allows the functional role of each brain-voxel to naturally emerge from the voxel-image cross-attention. We show the power of this approach to (i) combine data from multiple different subjects (a"Crowd of Brains") to improve each individual brain-encoding, (ii) quick&effective Transfer-Learning across subjects, datasets, and machines (e.g., 3-Tesla, 7-Tesla), with few training examples, and (iii) use the learned voxel-embeddings as a powerful tool to explore brain functionality (e.g., what is encoded where in the brain).

Problem

• Develops a Universal Brain-Encoder for diverse fMRI datasets.
• Enables cross-subject and cross-machine transfer learning efficiently.
• Explores brain functionality using learned voxel-embeddings.

Innovation

• Universal Brain-Encoder for multi-subject fMRI data
• Voxel-centric architecture with cross-attention mechanism
• Transfer-Learning across subjects, datasets, and machines