Spatial Normalization for Cross-Domain Retinal Layer Segmentation in Optical Coherence Tomography

2026-07-17

AI Summary

This study addresses the performance degradation of retinal layer segmentation in optical coherence tomography (OCT) caused by variations in acquisition protocols and population heterogeneity. To mitigate geometric domain shift, the authors propose a fovea-centric spatial normalization preprocessing framework that aligns OCT volumes to a unified anatomical reference space. This work pioneers the adaptation of spatial standardization concepts from neuroimaging to OCT segmentation and introduces a novel topology violation metric that operates without ground truth, alongside an en-face qualitative assessment method based on retinal thickness. By integrating multiple state-of-the-art segmentation architectures with multi-level evaluation strategies, the approach substantially enhances model robustness, generalizability, and clinical interpretability across heterogeneous datasets, offering a reliable tool for biomarker extraction in neurodegenerative disease research.

Abstract

Retinal layer segmentation in Optical Coherence Tomography (OCT) is a fundamental step for extracting quantitative biomarkers of retinal structure. Indeed, there is a growing interest in the analysis of OCTs in the context of neurodegenerative diseases. However, segmentation remains challenging due to speckle noise, shadowing artifacts, low contrast between adjacent layers, anatomical variability across subjects, and domain shifts arising from different acquisition protocols and clinical populations. While deep learning methods have achieved remarkable performance, their robustness and generalization across heterogeneous datasets remain limited. In this work, we investigate the role of spatial normalization as a preprocessing strategy to mitigate geometric domain shifts and improve the consistency of retinal layer segmentation. Inspired by standard practices in neuroimaging, we introduce a fovea-centered normalization framework that aligns OCT volumes into a common anatomical reference. We perform a comprehensive evaluation of state-of-the-art deep learning architectures. To provide a comprehensive assessment of segmentation quality, we combine conventional overlap-based metrics at B-scan level with topology-aware metrics at A-scan level and thickness-based measures at the en-face level. In cases where a ground truth is not available, we propose topology violation quantitative metrics that do not require ground truth annotations and a thickness-based qualitative assessment that captures structural consistency and clinically relevant patterns at the en-face level. The results demonstrate the importance of spatial normalization in OCT segmentation pipelines toward the development of robust and clinically meaningful retinal analysis tools, enabling reliable biomarker extraction and downstream computational analysis in neurodegenerative research.

Problem

• retinal layer segmentation
• optical coherence tomography
• domain shift
• spatial normalization
• cross-domain

Innovation

• spatial normalization
• cross-domain segmentation
• fovea-centered alignment
• topology-aware metrics
• OCT biomarker extraction