Sample complexity bounds for categorical Markov random fields via Discrete Diffusions

2026-10-01

AI Summary

This study addresses the absence of end-to-end sample complexity guarantees for sampling from high-dimensional categorical distributions. Focusing on locally dependent discrete data, it proposes a discrete diffusion sampling method based on uniform noise injection. The core innovation lies in introducing a pinning decomposition of the discrete score, which enables weight-sharing score networks and flexible discretization at inference time, while yielding optimal sample complexity bounds. By integrating low-order Markov random fields, neural score learning, and $\tau$-leaping techniques, the proposed framework provides both theoretical rigor and practical efficiency. Experiments on Potts and Ising models demonstrate that weight-sharing networks significantly outperform fully connected architectures when sampling long sequences.

Abstract

Many applications in statistics, economics, and physics require sampling from high-dimensional categorical distributions with local dependence structures. Examples include finite memory language models, Ising and Potts systems in statistical physics and protein folding, etc. In modern machine learning, discrete diffusions have emerged as a flexible approach for sampling such data, with strong empirical performance. Motivated by this, we develop learning methods with end-to-end sample complexity bounds for discrete diffusion with uniform noising under local dependence, which we model through low order Markov random fields (MRFs). Our main technical insight is a new \emph{pinning decomposition} of the discrete score. It shows that unlike in continuous diffusions, the score decomposes into components where the dependence on time separates multiplicatively from the dependence on the target. Building on this decomposition, we propose a \emph{weight-sharing neural score learner} and combine it with $τ$-leaping to obtain an end-to-end sampling procedure. Rather than treating score-learning error as a black-box input, as is common in existing sampling analyses, we study the score learning error from finite data and derive optimal sampling guarantees with explicit dependence on the vocabulary size, the interaction order of the MRF, and the sample size. Moreover, our strategy trains a single score network across uniform noise levels while leaving the sampling discretization to be chosen at inference-time. This allows the same trained model to trade accuracy for computational cost as inference-time budgets vary. Numerical experiments on Potts, Ising, and tree-structured models show that weight-sharing score networks outperform fully connected ones for sampling long sequences.

Problem

• discrete diffusion
• sample complexity
• categorical distributions
• Markov random fields
• score learning

Innovation

• Discrete Diffusion
• Pinning Decomposition
• Weight-sharing Neural Score Learner
• Sample Complexity Bounds
• Markov Random Fields