Discrete Distribution Networks

2023-12-29

arXiv.org

AI Summary

Efficient approximation of continuous/discrete data distributions and discrete sample generation remain challenging, particularly without continuous latent variables or explicit density estimation. Method: This paper proposes the Discrete Distribution Network (DDN), a novel generative framework based on hierarchical conditional feedback sampling. DDN generates discrete outputs layer-by-layer, with each layer conditioned on the previous layer’s optimal output, enabling progressive refinement. It operates without continuous latent variables or explicit probability density modeling, supporting zero-shot conditional generation and one-dimensional latent representations. Contribution/Results: DDN introduces the first hierarchical discrete distribution modeling paradigm, integrating ground-truth-guided selection and discrete multi-output generation. Experiments on CIFAR-10 and FFHQ demonstrate exponential improvement in generation quality with increasing layers, achieving both high fidelity and structural interpretability.

Technology Category

• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Machine Learning: Deep Generative Models & Autoencoders
• Natural Language Processing: Generation

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Social Networks and Social Media: Generative AI / large language models and their impact on social systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

We introduce a novel generative model, the Discrete Distribution Networks (DDN), that approximates data distribution using hierarchical discrete distributions. We posit that since the features within a network inherently capture distributional information, enabling the network to generate multiple samples simultaneously, rather than a single output, may offer an effective way to represent distributions. Therefore, DDN fits the target distribution, including continuous ones, by generating multiple discrete sample points. To capture finer details of the target data, DDN selects the output that is closest to the Ground Truth (GT) from the coarse results generated in the first layer. This selected output is then fed back into the network as a condition for the second layer, thereby generating new outputs more similar to the GT. As the number of DDN layers increases, the representational space of the outputs expands exponentially, and the generated samples become increasingly similar to the GT. This hierarchical output pattern of discrete distributions endows DDN with unique properties: more general zero-shot conditional generation and 1D latent representation. We demonstrate the efficacy of DDN and its intriguing properties through experiments on CIFAR-10 and FFHQ. The code is available at https://discrete-distribution-networks.github.io/

Problem

• Approximates data distribution using hierarchical discrete distributions
• Generates multiple discrete samples to fit target distributions
• Enhances sample similarity to Ground Truth through layered refinement

Innovation

• Hierarchical discrete distributions approximate data
• Multi-layer feedback refines output similarity
• Exponential representational space expansion enhances accuracy