Depth as Time in One-Step Generative Models

2026-10-02

AI Summary

This study investigates how single-step generative models reconstitute the multi-step denoising trajectories of diffusion processes, a computational mechanism that has remained poorly understood. Through intermediate-layer probing, flow matching theoretical analysis, and single-block time-conditioned training, this work provides the first empirical evidence that inter-layer denoising and renoising phenomena unfold along network depth within single-step models. Accordingly, it proposes a "depth-as-time" mechanism and reveals its dependence on transport task design. Leveraging these insights, this paper achieves extreme parameter compression by reducing the MeanFlow SiT-L/2 model 16.6-fold to a single time-conditioned module while preserving high-fidelity generation capabilities.

Abstract

The recent wave of one-step generative models, which compress the multi-step trajectory of diffusion via either distillation or learned flow maps, has reached an inflection point where they can generate high-quality images. Here, we ask a natural question that follows from these advances: what happens to the denoising trajectory of multi-step diffusion when generation is compressed into a single forward pass? We offer an empirical observation we call \textit{depth as time}: the denoising computation that multi-step diffusion performs across sampling steps appears to unfold across the depth of a single forward pass, and can be recovered by decoding intermediate layers with the model's own output head. Most interestingly, we show that this depthwise computation depends on the transport task a flow map is trained to solve. The most surprising case is MeanFlow, where probing shorter transport intervals reveals both denoising and renoising within a single network evaluation. In contrast, generators trained without a time-indexed transport task, such as drifting models, do not exhibit the same depthwise denoising. Consequently, we show that models that exhibit the depthwise denoising phenomenon are more compressible across the layerwise computation: a MeanFlow \texttt{SiT-L/2} model can be compressed by $16.6\times$ in parameters into a single time-conditioned block. We offer an explanation for this denoise-then-renoise behavior and show that, when we treat the layerwise computation explicitly as a flow, a single time-conditioned block can be trained to denoise across layers, compressing a MeanFlow \texttt{SiT-L/2} model by $16.6\times$ in parameters. Together, these results suggest that the temporal computation of diffusion is not eliminated by one-step generation, but reorganized across network depth.

Problem

• one-step generative models
• diffusion trajectory
• depth as time
• denoising computation
• MeanFlow

Innovation

• One-Step Generative Models
• Depth as Time
• MeanFlow
• Model Compression
• Flow Matching