Pixel-Space Diffusion Transformers

2026-07-20

AI Summary

This work addresses the limitations of latent diffusion models, which suffer from detail loss and misalignment between representation and generation objectives due to fixed visual encoders. To overcome these issues, the authors propose an end-to-end pixel-space diffusion Transformer framework that operates directly in the pixel domain without relying on VAE compression. The approach integrates a continuous generation mechanism within a unified multimodal Transformer architecture, sharing a common token space for both images and text. By carefully optimizing noise scheduling, loss weighting, and model scaling strategies, the method significantly enhances fine-grained detail fidelity in high-resolution image synthesis. This paradigm offers a promising direction toward building integrated multimodal vision foundation models capable of both generative and perceptual tasks.

Abstract

Latent diffusion models (LDMs) enable efficient high-resolution image synthesis by denoising in a VAE-compressed latent space. However, fixed visual tokenizers can discard fine textures and structural details, while separate representation and diffusion training creates a mismatch between reconstruction and generation objectives. These limitations have renewed interest in pixel-space diffusion, which models raw pixels directly, removes the VAE bottleneck, and supports end-to-end optimization. This formulation better matches the demands of high-fidelity generation but introduces challenges in high-dimensional modeling, including noise scheduling, loss weighting, token efficiency, and scalable architecture design. Pixel-space modeling also offers a promising basis for unified multimodal systems: raw pixels, text, and task conditions can be represented in a shared token space and jointly processed by a single Transformer, narrowing the gap between visual understanding and generation. This paper reviews Pixel-Space Diffusion Transformers (pDiTs) from the perspectives of model architecture, continuous generative mechanisms, and unified multimodal modeling. We summarize representative methods, identify key technical challenges, and discuss future directions toward high-fidelity, end-to-end vision foundation models that integrate generation and understanding.

Problem

• pixel-space diffusion
• high-fidelity generation
• multimodal modeling
• diffusion transformers
• end-to-end optimization

Innovation

• Pixel-space diffusion
• Diffusion Transformers
• End-to-end optimization
• Unified multimodal modeling
• Vision foundation models