ItDPDM: Information-Theoretic Discrete Poisson Diffusion Model

2025-05-08

AI Summary

Existing discrete sequence generation models rely on continuous embeddings and variational lower-bound (ELBO) approximations, leading to substantial bias in negative log-likelihood (NLL) estimation and slow convergence. This work introduces the first purely discrete-domain Poisson diffusion model, eliminating continuous embeddings and ELBO approximations entirely. Grounded in rigorous information-theoretic analysis, we derive an exact equivalence between the proposed Poisson reconstruction loss (PRL) and the true NLL. The Poisson diffusion process is formulated as a discrete-time Markov chain, and we design dedicated architectures for symbolic music (Lakh MIDI) and image token generation (CIFAR-10). Experiments demonstrate up to an 80% reduction in NLL and significantly accelerated training convergence. These results validate both the theoretical soundness and practical efficiency of discrete-domain diffusion modeling.

Technology Category

• Computer Vision: Diffusion Models for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Existing methods for generative modeling of discrete data, such as symbolic music tokens, face two primary challenges: (1) they either embed discrete inputs into continuous state-spaces or (2) rely on variational losses that only approximate the true negative log-likelihood. Previous efforts have individually targeted these limitations. While information-theoretic Gaussian diffusion models alleviate the suboptimality of variational losses, they still perform modeling in continuous domains. In this work, we introduce the Information-Theoretic Discrete Poisson Diffusion Model (ItDPDM), which simultaneously addresses both limitations by directly operating in a discrete state-space via a Poisson diffusion process inspired by photon arrival processes in camera sensors. We introduce a novel Poisson Reconstruction Loss (PRL) and derive an exact relationship between PRL and the true negative log-likelihood, thereby eliminating the need for approximate evidence lower bounds. Experiments conducted on the Lakh MIDI symbolic music dataset and the CIFAR-10 image benchmark demonstrate that ItDPDM delivers significant improvements, reducing test NLL by up to 80% compared to prior baselines, while also achieving faster convergence.

Problem

• Directly models discrete data without continuous embedding
• Eliminates approximate losses with exact Poisson reconstruction
• Improves test NLL by 80% over prior baselines

Innovation

• Direct discrete state-space modeling via Poisson diffusion
• Novel Poisson Reconstruction Loss for exact likelihood
• Eliminates approximate evidence lower bounds