Foundation VAEs for 3D CT Reconstruction, Augmentation, and Generation

2026-05-29

AI Summary

This work addresses the high training cost and limited generalizability of conventional CT-specific variational autoencoders (VAEs) across heterogeneous devices, protocols, and disease scenarios. It proposes, for the first time, a fine-tuning-free paradigm for medical VAEs by directly freezing both the encoder and decoder of a general-purpose Foundation VAE pretrained on natural images and videos, leveraging it as a unified interface for CT reconstruction, enhancement, and generation. A conditional latent diffusion model is integrated within the shared latent space to enable multitask collaboration. Experiments demonstrate significant performance gains: CT reconstruction improves surface segmentation accuracy (NSD) by 3.9%; generation tasks achieve a 3.9% reduction in FVD and a 36.2% increase in CT CLIP score; and fidelity across 18 disease categories shows an average AUC improvement of 2.76%.

Abstract

Variational autoencoders (VAEs) compress high resolution CT volumes into compact latents while preserving clinically relevant structure. However, training CT-specific VAEs from scratch or heavily fine-tuning them incurs substantial computational and engineering cost, and often degrades under heterogeneous scanners, protocols, and diseases. This paper makes a progressive stride toward training-free medical VAEs by leveraging a critical observation: a single Foundation VAE, pretrained at scale on natural images and videos, can serve as a unified interface for CT Reconstruction, Augmentation, and Generation. With both encoder and decoder frozen, the Foundation VAE reconstructs CT volumes with preserved anatomy while suppressing acquisition noise; training segmentation models on these reconstructions improves surface accuracy by 3.9% NSD on average for pancreatic tumor and lung tumor. Within the same Foundation VAE latent space, a conditional latent diffusion model achieves 3.9% lower average FVD with 36.2% higher CT CLIP score, and improves multi-disease generation faithfulness across 18 types by 2.76% AUC. These results demonstrate Foundation VAEs as a practical interface for scalable CT representation reuse and faithful CT generation. Our code and demo are available at https://github.com/qic999/Foundation-VAE.

Problem

• Foundation VAE
• 3D CT Reconstruction
• Medical Image Generation
• Latent Representation
• Cross-domain Generalization

Innovation

• Foundation VAE
• training-free
• CT reconstruction
• latent diffusion
• medical image generation