Diffusable Latents from Structure-Agnostic Distillation

2026-09-30

AI Summary

This study addresses the limitation of standard knowledge distillation, which binds latent layouts to the teacher model and thereby constrains the convergence speed and generation quality of diffusion models. To overcome this, we propose a structure-agnostic distillation framework that replaces position-wise matching with image-level pooled feature alignment. We theoretically demonstrate that dense spatial constraints are unnecessary, and instead employ first-order and relational pooling objectives to optimize the autoencoder bottleneck for compatibility with the diffusion process. This approach facilitates cross-modal transfer and accommodates heterogeneous latent shapes, yielding faster convergence and superior sample quality. Notably, our method successfully transfers knowledge from text encoders to image autoencoders, demonstrating its broad applicability in enhancing generative modeling through flexible, layout-independent distillation.

Abstract

Distilling pretrained foundation models into an autoencoder bottleneck improves latent diffusability, enabling diffusion models to converge faster and reach higher sample quality. Standard distillation aligns the latent at each position to a co-located teacher feature, tying the latent layout to the teacher's. We show this constraint is unnecessary: aligning a single pooled image-level descriptor to the teacher's performs as well as or slightly better than dense position-wise distillation. We compare first-order and relational pooled objectives across latent shapes and teacher modalities. First-order matching extends naturally to 1D token-sequence latents and across modalities, where distilling a text encoder into an image autoencoder still improves diffusability; a relational objective based only on each image's nearest neighbours improves it as well. Code and blog post are available at https://github.com/AdrienRR/structure-agnostic-distillation and https://kyutai.org/blog/2026-09-28-structure-agnostic-distillation/.

Problem

• Knowledge Distillation
• Latent Diffusion Models
• Autoencoder
• Foundation Models
• Structure-Agnostic

Innovation

• Structure-Agnostic Distillation
• Latent Diffusability
• Autoencoder Bottleneck
• Pooled Descriptor Alignment
• Cross-Modal Distillation