On the Hardness of Approximating Distributions with Probabilistic Circuits

2025-06-02

AI Summary

This paper investigates the feasibility and complexity of approximating arbitrary probability distributions using probabilistic circuits (PCs) under small approximation errors, aiming to mitigate exponential size blowup caused by structural constraints such as decomposability and determinism. Method: Leveraging tools from computational complexity theory, *f*-divergence analysis, and polynomial-time reductions, the authors establish tight theoretical lower bounds on approximate representational complexity. Contribution/Results: The work delivers two key results: (i) it proves that achieving ε-approximation—in the sense of *f*-divergence—of any model with efficiently computable marginal distributions is NP-hard; and (ii) it reveals an exponential gap in approximation size between decomposable and deterministic PCs. These findings rigorously quantify the fundamental trade-off between structural constraints and approximation efficiency, providing an unattainable complexity benchmark for PC architecture design.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Probabilistic Programming
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

A fundamental challenge in probabilistic modeling is balancing expressivity and tractable inference. Probabilistic circuits (PCs) aim to directly address this tradeoff by imposing structural constraints that guarantee efficient inference of certain queries while maintaining expressivity. Since inference complexity on PCs depends on circuit size, understanding the size bounds across circuit families is key to characterizing the tradeoff between tractability and expressive efficiency. However, expressive efficiency is often studied through exact representations, where exactly encoding distributions while enforcing various structural properties often incurs exponential size blow-ups. Thus, we pose the following question: can we avoid such size blow-ups by allowing some small approximation error? We first show that approximating an arbitrary distribution with bounded $f$-divergence is $mathsf{NP}$-hard for any model that can tractably compute marginals. We then prove an exponential size gap for approximation between the class of decomposable PCs and additionally deterministic PCs.

Problem

• Balancing expressivity and tractable inference in probabilistic modeling
• Understanding size bounds in probabilistic circuits for efficient inference
• Investigating approximation to avoid exponential size blow-ups in distributions

Innovation

• Balancing expressivity and tractable inference
• Approximating distributions with bounded divergence
• Exponential size gap for approximation