Where Does Generative Difficulty Reside? An Empirical Study of Target Representations

2026-08-01

AI Summary

This work challenges the common assumption that different target representations are interchangeable in image generation, systematically investigating how representation choice affects generative difficulty. Within a unified masked autoregressive Rectified Flow framework, the authors evaluate raw pixels, SD-VAE latents, DINOv2 features, and MAE embeddings on ImageNet, analyzing their generation behavior and optimization characteristics. The study reveals that no single property—such as compressibility, reconstruction fidelity, dimensionality, or semantic clustering—sufficiently predicts generative performance. Instead, different representations redistribute difficulty across contextual modeling, per-token denoising, and control over the inference distribution: DINOv2 converges fastest but relies on a wide local denoiser, pixel-based training is slow and requires specialized configurations, MAE yields faithful reconstructions yet poor generative quality, and all representations exhibit markedly distinct trade-offs between precision and recall as well as responsiveness to guidance.

Abstract

The target representation defines the distribution an image generator must learn, yet it is often treated as an interchangeable interface. This assumption is particularly questionable for continuous masked generators, which combine contextual inference from visible tokens with conditional modeling of each missing token. We study raw pixels, SD-VAE latents and DINOv2 as well as MAE representation-autoencoder features within a unified masked autoregressive rectified-flow model. Under a shared ImageNet training budget, these spaces exhibit distinct optimization and inference regimes. DINOv2 converges fastest in both iterations and computation but benefits strongly from a wider local denoiser and direct context fusion. Pixels optimize substantially more slowly and require a different prediction, masking, and guidance configuration. MAE reconstructs images more faithfully and exhibits clear semantic clustering, yet produces generations substantially worse than DINOv2. The representations also respond differently to classifier-free guidance and occupy distinct precision-recall trade-offs. Together, our results show that compression, reconstruction fidelity, token dimensionality, and visible semantic clustering do not individually predict generative behavior. Instead, target representations redistribute difficulty across contextual modeling, per-token denoising, and inference-time distributional control.

Problem

• target representation
• generative difficulty
• masked image generation
• representation learning
• image synthesis

Innovation

• target representation
• masked generative modeling
• DINOv2
• rectified flow
• generative difficulty