Should We Skip Diffusion?

2026-10-04

AI Summary

This study addresses the limitation that residual connections in diffusion model encoders hinder the separation of feature abstraction hierarchies. To overcome this, we propose the Decoupled Diffusion Transformer (DDT-RFE), which innovatively removes residual connections from self-attention and MLP modules to enhance feature decoupling and abstraction capabilities. Furthermore, a cross-layer multi-scale feature fusion mechanism is introduced to preserve fine-grained details for the decoder. Experimental results demonstrate that DDT-RFE outperforms baseline models across visual tasks such as image classification and ImageNet generation. Notably, the proposed approach effectively reduces the required number of encoder blocks while significantly lowering FID scores, achieving superior generative performance with improved architectural efficiency.

Abstract

Diffusion models learn semantic representations while generating images. In the Decoupled Diffusion Transformer (DDT), a condition encoder provides features that guide a velocity decoder in denoising. To enable effective denoising at all noise levels, these features must capture both high-level abstract structures and low-level details. However, skip/residual connections in the encoder allow shallow features to bypass successive transformations, which may limit progressive abstraction, or at least make it difficult to disentangle different levels of abstraction. We propose DDT-RFE, which removes the residual connections around the Self-Attention and MLP operations in each encoder block while maintaining stable training. To retain the information that abstraction discards but that the decoder still needs, we fuse the input patch embedding with intermediate and final encoder features to form the encoder output. The decoder thus has access to information from multiple encoder depths, while each encoder block is able to learn more abstract representations. DDT-RFE achieves overall improvements over DDT across visual understanding tasks, including image classification, semantic segmentation, object discovery, and semantic correspondence, while using fewer encoder blocks. It also achieves a lower FID for image generation on ImageNet.

Problem

• Diffusion Transformer
• Residual Connections
• Feature Abstraction
• Semantic Representation
• Decoupled Diffusion

Innovation

• Decoupled Diffusion Transformer
• Residual Connection Removal
• Feature Fusion
• Semantic Representation
• Visual Understanding