Towards Stabilized and Efficient Diffusion Transformers through Long-Skip-Connections with Spectral Constraints

2024-11-26

AI Summary

To address error accumulation and failure in cached inference of diffusion Transformers (DiTs) caused by unstable dynamic feature propagation, this paper proposes a Spectrally Constrained Long Skip Connection (LSC) mechanism: it introduces U-Net–style long skip connections into the DiT architecture for the first time, coupled with spectral norm regularization to ensure dynamic stability during feature propagation; additionally, a static hierarchical caching strategy is designed to enable cross-timestep reuse of deep features. The method preserves generation quality while significantly improving efficiency—accelerating training by 4.4× and speeding up convergence, and accelerating inference by 1.5–2×. Experiments demonstrate that our approach consistently outperforms existing DiT caching methods on both image and video generation tasks, establishing a new paradigm for efficient and stable diffusion modeling.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Diffusion Transformers (DiT) have emerged as a powerful architecture for image and video generation, offering superior quality and scalability. However, their practical application suffers from inherent dynamic feature instability, leading to error amplification during cached inference. Through systematic analysis, we identify the absence of long-range feature preservation mechanisms as the root cause of unstable feature propagation and perturbation sensitivity. To this end, we propose Skip-DiT, a novel DiT variant enhanced with Long-Skip-Connections (LSCs) - the key efficiency component in U-Nets. Theoretical spectral norm and visualization analysis demonstrate how LSCs stabilize feature dynamics. Skip-DiT architecture and its stabilized dynamic feature enable an efficient statical caching mechanism that reuses deep features across timesteps while updating shallow components. Extensive experiments across image and video generation tasks demonstrate that Skip-DiT achieves: (1) 4.4 times training acceleration and faster convergence, (2) 1.5-2 times inference acceleration without quality loss and high fidelity to original output, outperforming existing DiT caching methods across various quantitative metrics. Our findings establish long-skip connections as critical architectural components for training stable and efficient diffusion transformers.

Problem

• Addresses dynamic feature instability in Diffusion Transformers (DiT)
• Proposes Long-Skip-Connections (LSCs) to stabilize feature propagation
• Enables efficient cached inference without quality degradation

Innovation

• Enhanced DiT with Long-Skip-Connections for stability
• Spectral norm analysis stabilizes feature dynamics
• Efficient statical caching mechanism reuses deep features