IDEAL: In-DEpth ALignment Makes A Discrete Representation AutoEncoder

2026-06-09

AI Summary

Existing discrete representation autoencoders based on vision foundation models suffer from limited reconstruction fidelity due to their reliance solely on deep-layer features, which discards fine-grained details. This work proposes IDEAL, a novel framework that, for the first time, jointly aligns quantized tokens with both shallow (detail-preserving) and deep (semantic-rich) features of a pretrained vision foundation model. By integrating multi-level feature alignment and optimized quantization, IDEAL enables discrete representations that simultaneously achieve high-fidelity reconstruction and strong semantic coherence. Evaluated on ImageNet, the method attains an rFID of 0.61—improving by 0.28 over the previous state of the art—and achieves a new SOTA in autoregressive image generation with a gFID of 1.89.

Abstract

Built on pretrained vision foundation models (VFMs), representation autoencoders (RAEs) have recently emerged as a promising approach for constructing semantically rich latent spaces for image generation. However, their reconstruction quality often remains suboptimal, largely because deep VFM representations do not preserve sufficient fine-grained visual detail. This limitation becomes even more severe after discretization, where missing low-level information is difficult to recover. In fact, we observe that shallow VFM features retain considerably richer local appearance and structural detail, which complements the high-level semantics carried by deep features used in existing RAEs. Motivated by this complementary property, we propose Ideal, an In-depth Alignment framework for discrete representation autoencoding. By jointly aligning quantized tokens with both shallow and deep VFM features, Ideal enables the resulting discrete visual tokens to preserve both visual fidelity and rich semantics. Extensive experiments demonstrate that Ideal yields superior reconstruction performance, achieving 0.61 rFID on ImageNet and outperforming the previous best method by 0.28. When used for autoregressive image generation, Ideal further produces a gFID of 1.89, establishing a new state of the art for autoregressive image generation.

Problem

• representation autoencoder
• discrete representation
• visual fidelity
• semantic richness
• image reconstruction

Innovation

• discrete representation
• representation autoencoder
• vision foundation model
• feature alignment
• autoregressive generation