Pioneering 4-Bit FP Quantization for Diffusion Models: Mixup-Sign Quantization and Timestep-Aware Fine-Tuning

2025-05-27

AI Summary

Diffusion models face three key challenges in 4-bit floating-point (FP) quantization: asymmetric activation distributions, insufficient temporal dynamic modeling across denoising steps, and misalignment between quantization error minimization and the generative optimization objective. To address these, we propose Mixed-Sign Floating-Point (MSFP) quantization—introducing unsigned FP representation for non-negative activations; Temporal-Aware LoRA (TALoRA) to explicitly model inter-step dynamics during denoising; and Denoising Factor Alignment (DFA) loss, which jointly optimizes quantization error and generation quality. Our method enables end-to-end 4-bit FP joint optimization of both weights and activations. Experiments demonstrate that MSFP preserves over 99% of the original FID across multiple benchmarks, significantly outperforming existing 4-bit integer post-training quantization (PTQ) methods. To our knowledge, this is the first work achieving high-fidelity 4-bit floating-point quantization for diffusion models.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Quantum Machine Learning
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Model quantization reduces the bit-width of weights and activations, improving memory efficiency and inference speed in diffusion models. However, achieving 4-bit quantization remains challenging. Existing methods, primarily based on integer quantization and post-training quantization fine-tuning, struggle with inconsistent performance. Inspired by the success of floating-point (FP) quantization in large language models, we explore low-bit FP quantization for diffusion models and identify key challenges: the failure of signed FP quantization to handle asymmetric activation distributions, the insufficient consideration of temporal complexity in the denoising process during fine-tuning, and the misalignment between fine-tuning loss and quantization error. To address these challenges, we propose the mixup-sign floating-point quantization (MSFP) framework, first introducing unsigned FP quantization in model quantization, along with timestep-aware LoRA (TALoRA) and denoising-factor loss alignment (DFA), which ensure precise and stable fine-tuning. Extensive experiments show that we are the first to achieve superior performance in 4-bit FP quantization for diffusion models, outperforming existing PTQ fine-tuning methods in 4-bit INT quantization.

Problem

• Handling asymmetric activation distributions in 4-bit FP quantization
• Addressing temporal complexity in denoising during fine-tuning
• Aligning fine-tuning loss with quantization error effectively

Innovation

• Mixup-sign FP quantization for asymmetric activations
• Timestep-aware LoRA for precise fine-tuning
• Denoising-factor loss alignment for stable quantization