DSTAR: Accelerating Diffusion Transformers via Spatial and Temporal Redundancy Reduction

2026-07-17

AI Summary

This work addresses the high latency and energy consumption of Diffusion Transformers (DiT) caused by their iterative inference process, a challenge that existing acceleration methods struggle to overcome without sacrificing accuracy. To this end, the study introduces the first joint optimization framework targeting both spatial and temporal redundancies in DiT. It proposes a fine-grained mixed-precision quantization scheme based on differential activation and a sparse attention reuse mechanism, accompanied by a dedicated hardware accelerator. Evaluated across seven representative DiT models, the proposed approach achieves up to 7.33× lower latency and 41.89× lower energy consumption compared to an NVIDIA A100 GPU, and outperforms the state-of-the-art accelerator by 2.54× in speed and 3.68× in energy efficiency—all without any loss in model accuracy.

Abstract

Diffusion Transformers (DiTs) have been widely used in many tasks, including image synthesis, video generation, and content editing. However, their multi-iteration inference process leads to performance inefficiency and high energy consumption. Existing acceleration methods primarily focus on reducing temporal redundancy between adjacent timesteps, but often overlook the specific features of DiTs. As a result, these approaches either suffer from great accuracy degradation or fail to achieve high efficiency. We present DSTAR, a software-hardware co-design framework that accelerates DiT inference by reducing spatial and temporal redundancy. At the algorithmic level, DSTAR introduces a fine-grained mixed-precision quantization method for differential activations in linear operations, significantly increasing the proportion of low-bit computations. Additionally, DSTAR incorporates a sparse attention reuse mechanism to minimize redundant computation in attention layers. For architectural support, we design a specialized hardware accelerator which achieves high efficiency in both latency and energy consumption. Evaluation on seven typical DiTs demonstrates that DSTAR achieves up to 7.33x latency speedup and 41.89x energy savings compared to an NVIDIA A100 GPU, and achieves up to 2.54x latency speedup and 3.68x energy savings compared to SOTA accelerators, without accuracy degradation.

Problem

• Diffusion Transformers
• inference acceleration
• temporal redundancy
• spatial redundancy
• energy efficiency

Innovation

• Diffusion Transformers
• mixed-precision quantization
• sparse attention reuse
• spatial-temporal redundancy reduction
• hardware-software co-design