Self-Supervised Representation-Guided Generative Dataset Distillation

2026-08-04

AI Summary

This work addresses the limitation of existing dataset distillation methods, which overlook the geometric structure of representation spaces learned by self-supervised pre-trained models and thus struggle to adapt to modern vision systems with frozen encoders. To overcome this, we propose SRG, a novel framework that explicitly incorporates the geometric properties of self-supervised learning (SSL) representation spaces into the diffusion generation process. SRG employs a threefold guidance objective—class prototype alignment, inter-class separation, and intra-class distribution—combined with a staged denoising strategy to simultaneously preserve visual fidelity and precisely shape class-discriminative representations in distilled samples. Extensive experiments demonstrate that SRG consistently outperforms current generative baselines across multiple datasets and IPC settings, while also exhibiting strong cross-encoder transferability.

Abstract

Dataset distillation compresses a large training set into a compact synthetic set while retaining its downstream utility. Most existing methods target randomly initialized networks, whereas modern vision systems often adapt frozen pretrained encoders with lightweight modules. Distilled samples should therefore preserve the discriminative geometry of the pretrained representation space, which existing generative objectives do not explicitly consider. We propose self-supervised representation-guided generative dataset distillation (SRG), a framework that translates the SSL geometry into diffusion guidance. Specifically, SRG constructs class-wise prototypes from real-image SSL representations and performs guidance through three SSL-space objectives for prototype alignment, inter-class discrimination, and intra-class assignment. During diffusion sampling, it adopts a stage-wise guidance strategy: early denoising is anchored to the latent of the real image whose SSL representation is nearest to the assigned prototype, whereas later denoising is guided by the SSL-space objectives. This division preserves the visual realism provided by the generative prior while progressively steering samples toward representative and class-discriminative regions of the SSL representation space. SRG consistently outperforms the evaluated generative baselines across multiple datasets and IPC settings. A cross-encoder evaluation further indicates transfer across pretrained representation spaces. These results demonstrate the effectiveness of representation-guided generation for dataset distillation with pretrained SSL models.

Problem

• dataset distillation
• self-supervised learning
• pretrained representations
• generative models
• representation geometry

Innovation

• self-supervised learning
• dataset distillation
• diffusion guidance
• representation space
• generative modeling