Masked and Predictive Self-Supervised Foundation Models for 3D Brain MRI

2026-06-11

AI Summary

This work addresses the lack of systematic investigation into self-supervised foundation models for MRI-based disease detection, a domain where prior efforts have predominantly focused on dense prediction tasks such as segmentation. The study systematically compares two prominent paradigms—Masked Autoencoders (MAE) and Joint-Embedding Predictive Architectures (JEPA)—proposing a spectral-domain reconstruction loss to enhance MAE’s sensitivity to fine anatomical structures and introducing variance–covariance regularization in JEPA to encourage decorrelated latent representations. For the first time, it reveals the critical alignment between self-supervised pretraining objectives and the structural characteristics of downstream discriminative signals, enabling contrast-agnostic pretraining on single-contrast, heterogeneous 3D brain MRI without modality concatenation. Experiments across five disease detection tasks demonstrate that spectrally supervised MAE achieves superior performance, underscoring the importance of aligning pretraining objectives with downstream task structure.

Abstract

Self-supervised foundation models have shown strong promise in medical imaging. However, existing MRI foundation-model studies have primarily emphasized segmentation and dense prediction tasks, while systematic investigation of self-supervised foundation models for MRI-based disease detection remains limited. In this work, we investigate two major self-supervised pretraining paradigms for MRI-based disease detection: reconstruction-based learning via Masked Autoencoders (MAE) and predictive representation learning via Joint Embedding Predictive Architectures (JEPA). We study the role of auxiliary objectives by introducing a novel spectral-domain reconstruction loss for MAE to enhance sensitivity to fine-grained anatomical structure, and by integrating variance--covariance regularization (VCR) within our JEPA framework to encourage decorrelated latent representations. Our models are pretrained on heterogeneous single-contrast MRI volumes in a contrast-agnostic setting, without modality concatenation. Across five downstream disease detection tasks, our results highlight the importance of self-supervised objective design for medical foundation model pretraining, demonstrating that the downstream benefit of each objective is determined by its relevance to the task's structure. Specifically, spectral regularization yields the largest improvements when the downstream discriminative signal is characterized by strong high-frequency anatomical structures, while covariance regularization is most beneficial when discriminative information spans multiple decorrelated feature dimensions. MAE with spectral-domain supervision consistently achieves superior downstream performance for MRI-based disease detection. These findings suggest that self-supervised objectives in medical imaging encode specific biases, and their downstream benefit is fundamentally conditioned on the task's structure.

Problem

• self-supervised learning
• MRI-based disease detection
• foundation models
• medical imaging
• 3D brain MRI

Innovation

• Masked Autoencoder
• Joint Embedding Predictive Architecture
• spectral-domain reconstruction
• variance-covariance regularization
• self-supervised foundation model