SPAE: Spectrally Guided Autoencoder for Pretrained Visual Latents

2026-08-02

AI Summary

This work addresses the challenge that latent representations from existing Vision Foundation Models (VFMs) are ill-suited for generative tasks, primarily due to spectral mismatch in high-frequency components between latents produced by Diffusion Transformers (DiTs) and encoder outputs, as well as entangled semantic and high-frequency information across channel dimensions. To resolve this, the authors propose the SPAE framework, which employs a compact bottleneck autoencoder to extract stable semantic features while suppressing high-frequency noise. By integrating channel-wise spectral analysis with a masking strategy, SPAE effectively disentangles semantic content from high-frequency details. This approach significantly enhances the consistency between DiT-generated and encoder-derived latents, achieving a favorable trade-off among image generation quality, reconstruction fidelity, and visual understanding capability, thereby mitigating spectral mismatch in the latent space.

Abstract

Latents from vision foundation models (VFMs) are semantically rich and well suited for visual understanding. Recent representation autoencoder methods such as RAE have shown that they can provide promising latent spaces for image generation. However, VFM latents remain difficult to model directly: DiT-generated latents exhibit spectral mismatch with encoder latents, especially in high-frequency components. Our channel-wise spectral analysis further reveals that these high-frequency components are diffusely distributed across latent channels and entangled with semantic information, making the latent space difficult for DiT to model. To address these challenges, we propose SPAE, latent adaptation framework for generation. Specifically, SPAE employs a compact bottleneck to distill stable semantic information while suppressing high-frequency components, thereby improving the alignment between DiT-generated latents and encoder latents. In addition, we apply a channel-wise masking strategy to promote the decoupling of semantic information and high-frequency details across bottleneck channels. Experiments show that SPAE achieves a favorable balance among visual understanding, generation quality, and reconstruction fidelity.

Problem

• spectral mismatch
• visual foundation models
• latent representation
• high-frequency components
• semantic entanglement

Innovation

• spectral alignment
• latent adaptation
• channel-wise masking
• vision foundation models
• autoencoder