Real-time Reconstruction of Human Visual Perception from fMRI

2026-07-23

AI Summary

This study addresses the computational limitations of existing real-time fMRI analysis methods, which hinder the deployment of advanced visual perception decoding models in closed-loop neurofeedback applications. For the first time, the high-fidelity natural image reconstruction model MindEye2 is adapted to a strictly real-time processing framework by integrating it with the open-source RT-Cloud platform and an optimized fMRI pipeline, enabling visual image reconstruction within seconds per trial during actual scanning. The work demonstrates the feasibility of deploying state-of-the-art decoding models under stringent latency constraints without compromising reconstruction fidelity. Through simulation-based analyses, the study further identifies key factors underlying performance discrepancies between offline and real-time settings, offering a practical pathway toward low-latency, high-accuracy real-time brain–computer interfaces.

Abstract

Real-time closed-loop neurofeedback based on functional magnetic resonance imaging (fMRI) has led to important scientific and clinical advances. However, the sophistication of the analysis methods used in real-time fMRI lags behind the state-of-the-art in fMRI decoding, largely due to computational factors: Most advanced decoding pipelines do not fit within the envelope of real-time processing, where the analysis needs to be conducted in a matter of seconds and without leveraging data acquired later in the session. Here, we present a real-time compatible adaptation of a computationally intensive state-of-the-art pipeline for reconstructing perceived natural images (MindEye2), and we demonstrate that reliable fine-grained decoding is still achievable in this setting. Using RT-Cloud, an open-source, scalable cloud-based platform, we performed a real-time scan where we decoded single-trial visual perception within seconds after an image was shown to the participant. Finally, we use simulated analyses to document the factors driving changes in performance from offline to real-time analysis. This work serves as a proof-of-concept that it is feasible to deploy these powerful fMRI decoding pipelines in real-time analysis, paving the way for their use in brain-computer interfaces for scientific discovery and clinical treatment.

Problem

• real-time fMRI
• visual perception reconstruction
• neurofeedback
• brain-computer interface
• fMRI decoding

Innovation

• real-time fMRI
• neural decoding
• image reconstruction
• brain-computer interface
• RT-Cloud