Latent Space Is Not Flat: Rethinking Latent Structure for 3D Medical Image Synthesis

2026-09-26

AI Summary

This study addresses the decoding distortion caused by latent space geometric anisotropy in 3D medical image synthesis by proposing a Latent Structure Flow (LSF) method. LSF decouples latent states into structural and residual components to model global and local variations separately. By revealing that structural variations are governed by low-rank subspaces, we design an efficient architecture that modifies only the generator while freezing the pretrained encoder-decoder to reduce optimization complexity. Experimental results demonstrate that LSF consistently outperforms existing baselines across multimodal medical image synthesis and tumor inpainting tasks, achieving high-quality and structurally consistent 3D medical image generation.

Abstract

Latent generative models make 3D medical image synthesis computationally practical by generating in a compressed space. However, we show that the common flat Euclidean assumption induced by $\ell_2$ objectives is imprecise: latent-space geometry is so strongly anisotropic that equal-magnitude errors can produce drastically different decoded distortions. We further find that this anisotropy has a clear feature: sensitive variation concentrates in a low-rank subspace. The dominant low-rank components capture the overall structure, encoding long-range, spatially coordinated variation while remaining resistant to local noise. Its orthogonal residual, in contrast, mainly captures local and image-specific variation. Motivated by this asymmetry, we introduce Latent Structure Flow (LSF). At each block, LSF decomposes the latent state into structure and residual, models structural changes with global context, and predicts residual variation locally while preserving a direct path for the input structure. LSF changes only the generator, leaving the frozen codec and pointwise training objective unchanged. Across cross-modality synthesis and tumor inpainting tasks, LSF outperforms all compared baselines on both global and tumor-specific metrics, demonstrating the benefit of explicitly modeling latent-space structure for 3D medical image synthesis.

Problem

• 3D medical image synthesis
• latent space geometry
• anisotropy
• low-rank subspace
• latent generative models

Innovation

• Latent Space Geometry
• Anisotropy
• Low-rank Subspace
• Latent Structure Flow
• 3D Medical Image Synthesis