MindShot: Brain Decoding Framework Using Only One Image

2024-05-24

arXiv.org

AI Summary

Cross-subject variability in fMRI data and the high cost of acquiring paired fMRI–stimulus datasets severely hinder generalizable visual reconstruction in brain decoding. Method: We propose the first few-shot cross-subject visual reconstruction paradigm, featuring: (1) a lightweight hemodynamic response function (HRF)-based adapter that explicitly models and compensates for inter-subject neural response variability; (2) a Fourier-domain cross-subject supervision mechanism leveraging fMRI signals from other subjects to inject multi-scale biological priors; and (3) integration of neural encoding disentanglement with few-shot transfer learning. Contribution/Results: Using only one image–fMRI pair per subject, our method significantly outperforms single-subject baselines, achieving semantically faithful and structurally coherent reconstructions. This work establishes, for the first time, the feasibility of large-model-driven, low-data cross-subject brain decoding—demonstrating robust performance under extreme data scarcity.

Technology Category

• Computer Vision: Multi-modal Vision
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Multimodal Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Graph Algorithms and Modeling for the Web: Representation, reconstruction, and subgraph or motif discovery in Web-related graphs

Abstract

Brain decoding, which aims at reconstructing visual stimuli from brain signals, primarily utilizing functional magnetic resonance imaging (fMRI), has recently made positive progress. However, it is impeded by significant challenges such as the difficulty of acquiring fMRI-image pairs and the variability of individuals, etc. Most methods have to adopt the per-subject-per-model paradigm, greatly limiting their applications. To alleviate this problem, we introduce a new and meaningful task, few-shot brain decoding, while it will face two inherent difficulties: 1) the scarcity of fMRI-image pairs and the noisy signals can easily lead to overfitting; 2) the inadequate guidance complicates the training of a robust encoder. Therefore, a novel framework named MindShot, is proposed to achieve effective few-shot brain decoding by leveraging cross-subject prior knowledge. Firstly, inspired by the hemodynamic response function (HRF), the HRF adapter is applied to eliminate unexplainable cognitive differences between subjects with small trainable parameters. Secondly, a Fourier-based cross-subject supervision method is presented to extract additional high-level and low-level biological guidance information from signals of other subjects. Under the MindShot, new subjects and pretrained individuals only need to view images of the same semantic class, significantly expanding the model's applicability. Experimental results demonstrate MindShot's ability of reconstructing semantically faithful images in few-shot scenarios and outperforms methods based on the per-subject-per-model paradigm. The promising results of the proposed method not only validate the feasibility of few-shot brain decoding but also provide the possibility for the learning of large models under the condition of reducing data dependence.

Problem

• Reconstruct visual stimuli from fMRI brain signals
• Address individual differences in brain decoding
• Reduce data collection costs for clinical applications

Innovation

• Multi-subject pretraining via contrastive learning
• Fourier-based cross-subject knowledge distillation
• Few-shot decoding with reduced fMRI data requirements