Discrete Copula Diffusion

2024-10-02

arXiv.org

AI Summary

Discrete diffusion models for sequence modeling (e.g., text, DNA) typically require hundreds of denoising steps—far more than their continuous counterparts—due to the neglect of inter-variable dependencies at each step, resulting in slow convergence. This work identifies such dependency ignorance as the fundamental bottleneck. We propose a Copula-augmented framework that decouples autoregressive Copulas from discrete diffusion and couples them *during* denoising, explicitly modeling the joint distribution of output variables at every step—without joint training or fine-tuning. Crucially, our approach preserves the original diffusion architecture and injects dependencies solely via posterior recalibration. Experiments on unconditional and conditional text generation show that our method achieves superior performance in only 8–32 denoising steps—just 1/8 to 1/32 of those required by standard discrete diffusion—yielding 8×–32× speedup and significantly outperforming both pure diffusion and pure Copula baselines.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

Discrete diffusion models have recently shown significant progress in modeling complex data, such as natural languages and DNA sequences. However, unlike diffusion models for continuous data, which can generate high-quality samples in just a few denoising steps, modern discrete diffusion models still require hundreds or even thousands of denoising steps to perform well. In this paper, we identify a fundamental limitation that prevents discrete diffusion models from achieving strong performance with fewer steps -- they fail to capture dependencies between output variables at each denoising step. To address this issue, we provide a formal explanation and introduce a general approach to supplement the missing dependency information by incorporating another deep generative model, termed the copula model. Our method does not require fine-tuning either the diffusion model or the copula model, yet it enables high-quality sample generation with significantly fewer denoising steps. When we apply this approach to autoregressive copula models, the combined model outperforms both models individually in unconditional and conditional text generation. Specifically, the hybrid model achieves better (un)conditional text generation using 8 to 32 times fewer denoising steps than the diffusion model alone. In addition to presenting an effective discrete diffusion generation algorithm, this paper emphasizes the importance of modeling inter-variable dependencies in discrete diffusion.

Problem

• Improves discrete diffusion models by capturing inter-variable dependencies.
• Reduces denoising steps significantly using copula models.
• Enhances text generation quality with fewer computational steps.

Innovation

• Incorporates copula model for dependency capture
• Reduces denoising steps significantly
• Enhances text generation quality and efficiency