SPARE: Structural Parameter-Free Affinity Regularization for Flow Matching

2026-08-03

AI Summary

This work addresses the slow convergence of denoising diffusion Transformers by proposing a parameter-free structural regularization method that circumvents reliance on external encoders and better preserves intrinsic data structure. The approach uniquely leverages pairwise similarity relationships among tokens in clean latent representations as a direct supervision signal and innovatively incorporates cross-image token pairs to capture spatial structure. Built upon the flow matching framework, it introduces a parameter-free affinity regularization term that jointly calibrates intra-layer and inter-sample relationships. Evaluated on ImageNet 256×256 with a SiT backbone, the method incurs only 0.08 GB additional GPU memory without introducing new parameters, achieving superior FID compared to all existing parameter-free approaches at 400K training iterations and reaching an FID of 1.90 at 1M iterations when combined with REPA.

Abstract

Denoising diffusion transformers achieve strong generation quality but converge slowly during training. Regularizing their internal representations has emerged as an effective accelerator, yet existing methods split into two families with complementary costs. Target-based methods strengthen representations by aligning them to external features, which requires an external encoder and a learnable projection head to bridge feature spaces. Target-free methods hold no reference at all, and can only repel the model's own features across samples or layers, discarding whatever structure the data contains. Prior work suggests that spatial structure, rather than global semantics, drives the gains of alignment. We therefore ask whether such structure can serve as a target directly, and whether it exists not only within an image but across images. Our key insight is that the clean data latent already carries this structure in the relations among its tokens, where a relation is the similarity between two tokens, a single scalar comparable across feature spaces without a projection head. We propose Structural Parameter-free Affinity Regularization (SPARE), a regularizer that matches the pairwise affinities of intermediate tokens to those of the clean latents. To exploit this structure fully, SPARE extends the matching to token pairs across images, precisely the pairs that prior target-free methods repel by default, and calibrates both relation types with a single learning objective. On ImageNet $256 \times 256$ with SiT backbones under matched 400K-iteration budgets, SPARE adds no encoder, head, or parameters and only 0.08 GB of training memory, yet attains the lowest FID among parameter-free regularizers in every tested setting, recovers 37 to 54\% of REPA's FID reduction, and improves over REPA when combined with it, reaching FID 1.90 under classifier-free guidance at 1M iterations.

Problem

• regularization
• diffusion transformers
• representation learning
• affinity matching
• parameter-free

Innovation

• affinity regularization
• parameter-free
• flow matching
• structural alignment
• cross-image token relations