On the role of non-linear latent features in bipartite generative neural networks

2025-06-12

AI Summary

This work identifies the fundamental mechanism underlying the degraded critical capacity of standard binary Restricted Boltzmann Machines (RBMs) when used as associative memories: anomalous phase transitions arising from overly simplistic hidden-unit priors and global connectivity. To address this, we propose an enhanced RBM architecture incorporating local biases and expressive priors—such as multi-state or ReLU-like distributions—for hidden units. Leveraging statistical physics and disordered-system theory, we systematically analyze the impact of diverse hidden-unit priors on phase-diagram structure and finite-temperature memory retrieval performance, using Monte Carlo simulations and energy landscape modeling. Results demonstrate that the proposed architecture restores a robust ordered retrieval phase, substantially increasing memory capacity and noise robustness. This study provides thermodynamically grounded theoretical principles and a practical architectural paradigm for designing interpretable latent spaces in generative models.

Technology Category

• Machine Learning: Probabilistic Circuits and Graphical Models
• Reasoning under Uncertainty: Relational Probabilistic Models
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web query analysis, representation and understanding
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

We investigate the phase diagram and memory retrieval capabilities of bipartite energy-based neural networks, namely Restricted Boltzmann Machines (RBMs), as a function of the prior distribution imposed on their hidden units - including binary, multi-state, and ReLU-like activations. Drawing connections to the Hopfield model and employing analytical tools from statistical physics of disordered systems, we explore how the architectural choices and activation functions shape the thermodynamic properties of these models. Our analysis reveals that standard RBMs with binary hidden nodes and extensive connectivity suffer from reduced critical capacity, limiting their effectiveness as associative memories. To address this, we examine several modifications, such as introducing local biases and adopting richer hidden unit priors. These adjustments restore ordered retrieval phases and markedly improve recall performance, even at finite temperatures. Our theoretical findings, supported by finite-size Monte Carlo simulations, highlight the importance of hidden unit design in enhancing the expressive power of RBMs.

Problem

• Study phase diagram and memory retrieval in RBMs
• Analyze impact of hidden unit priors on performance
• Improve RBM capacity via architectural modifications

Innovation

• Using non-linear latent features in RBMs
• Employing statistical physics analytical tools
• Modifying RBMs with local biases