BudgetPix: Compute-Adaptive Tokenization for Pixel-Space Image Diffusion

2026-10-08

AI Summary

This study addresses the computational redundancy and rigid inference budgets caused by static, uniform tokenization in pixel-space diffusion models by proposing an adaptive tokenization framework. The method introduces a variable-length sequence encoding scheme that combines entropy-guided quadtree partitioning with multi-scale embeddings to dynamically allocate computational resources according to local visual complexity. Furthermore, it designs an adaptive encoder, a scale-aware decoder, and a flexible training sampling schedule, enabling a single model to operate elastically across diverse computational budgets. Experimental results demonstrate that the proposed framework matches the image quality of mainstream models while consuming only 25% of the original computational cost, significantly optimizing the trade-off between generation quality and computational efficiency.

Abstract

Most image generation models rely on uniform tokenization, allocating the exact same computational budget to equally-sized image patches. This static paradigm cannot adapt to different resource constraints at inference time, and yields suboptimal quality-cost tradeoff by devoting the same effort to both plain backgrounds and intricate details. We propose BudgetPix, an adaptive tokenization framework that dynamically allocates compute based on visual complexity and spatial layout, enabling flexible computational budgeting at inference time. BudgetPix comprises three key components: (1) an adaptive encoder that maps a fixed-size image to a variable-length token sequence using an entropy-guided quadtree alongside a multi-scale patch embedder; (2) a scale-aware decoder reconstructs fixed-resolution images from multi-scale token sets; and (3) a flexible training and sampling schedule that enables pixel-space denoisers to operate across variable token counts. BudgetPix seamlessly integrates with existing pixel-space diffusion architectures, enabling a single checkpoint to be operated at a wide range of compute budgets. Evaluated on text-to-image generation, BudgetPix matches the fidelity of MiniT2I-L at $512^2$ and PixelDiT at $1024^2$ using just 25% of the original compute budget. In class-conditional generation using a MeanFlow backbone, BudgetPix requires merely 60% of the full compute budget to produce images with near-zero quality degradation, observing a marginal 0.8-point increase in FID. Comprehensive assessments by human and VLM judges confirm that BudgetPix establishes a significantly improved quality-efficiency tradeoff over prior budget-adaptive baselines. More details are available at our project page: https://karaozgur.com/BudgetPix

Problem

• image generation
• uniform tokenization
• compute budget
• quality-cost tradeoff
• pixel-space diffusion

Innovation

• Adaptive Tokenization
• Pixel-Space Diffusion
• Compute Budgeting
• Entropy-Guided Quadtree
• Multi-Scale Patch Embedder