fMRI2Face: A Full-HD fMRI-Video Dataset and Geometry-Guided Neural Decoding Framework for Dynamic Human Face Reconstruction

2026-07-24

AI Summary

This study addresses the limitations of existing fMRI-based facial decoding methods, which struggle to simultaneously recover identity-specific features and dynamic facial changes due to low spatial resolution and a lack of controllable, high-quality neural imaging data. To overcome these challenges, the authors introduce fMRI-Face, the first paired fMRI dataset aligned with full high-definition digital facial videos, and propose fMRI2Face, a geometry-guided neural video decoding framework. fMRI2Face integrates a dual-path architecture that combines Brain-derived Appearance Context for identity extraction with Morphable 3D Facial Control to model expressions and head poses, leveraging Neural-Controlled Video Diffusion to synthesize high-fidelity dynamic facial videos. Experimental results demonstrate that the proposed method significantly outperforms current baselines in reconstruction fidelity, identity preservation, geometric accuracy, and motion consistency.

Abstract

Reconstructing dynamic human faces from brain activity provides a powerful way to study how the mind perceives identity, expression, and facial motion. However, progress in fMRI-based face decoding has been limited by scarce controlled, high-resolution neural datasets and by methods that struggle to recover both identity-specific appearance and time-varying facial dynamics. We present fMRI-Face, the first fMRI dataset paired with controllable full-HD digital human facial videos rendered at 1920$\times$1080 resolution. During scanning, participants watched photorealistic, background-free facial videos with controlled identity, expression, and head pose, while fMRI activity was recorded. The resulting dataset contains 62,856 paired fMRI-video samples, providing a structured resource for studying dynamic face perception and reconstruction. Building on this dataset, we propose fMRI2Face, a geometry-guided neural video decoding framework for reconstructing facial videos from fMRI signals. fMRI2Face derives two complementary neural controls from brain activity: Brain-derived Appearance Context, which captures global identity-related visual attributes, and Morphable 3D Facial Control, which provides explicit geometry-aware guidance for pose, expression, and non-rigid facial dynamics. These controls are integrated through Neural-Controlled Video Diffusion with auxiliary latent completion, enabling high-fidelity facial video reconstruction directly from brain activity. Experiments show that fMRI2Face consistently improves reconstruction fidelity, identity preservation, facial geometry, and motion consistency over representative neural decoding baselines. Together, fMRI-Face and fMRI2Face establish a controlled platform for studying dynamic face perception and provide a new benchmark for fMRI-based digital human reconstruction.

Problem

• fMRI
• face reconstruction
• dynamic facial video
• neural decoding
• digital human

Innovation

• fMRI-based video reconstruction
• geometry-guided neural decoding
• dynamic face perception
• neural-controlled diffusion
• full-HD brain-to-face dataset