Convergent Evolution in Neural Representation Space: Emergent Order in Deep Belief Networks

2026-08-06

AI Summary

This study investigates whether deep belief networks (DBNs) trained entirely in an unsupervised manner can spontaneously develop internal representational structures aligned with data categories. Through comprehensive analysis of layer-wise representations on datasets such as MNIST—employing generalized discriminative values (GDV), supervised probing, reconstruction abstraction metrics, effective dimensionality estimation, and free sample generation—the work systematically demonstrates that deeper layers exhibit emergent clustering by class, progressive prototypicality, and markedly enhanced class separability. This clustering effect arises from the learned feature structure rather than random initialization or trivial transformations, providing the first empirical evidence of self-organized emergence of categorical structure in unsupervised deep models.

Abstract

Deep Belief Networks (DBNs) learn hierarchical generative models without class supervision. Here, we ask whether this purely unsupervised process nevertheless organizes internal representations according to the unknown data classes. We analyze successive layers of DBNs trained on MNIST, Fashion-MNIST, and KMNIST using the Generalized Discrimination Value (GDV), supervised probes applied only after training, a reconstruction-based measure of abstraction distance, effective dimensionality, and free sample generation. Remarkably, class-specific clustering generally increases with depth across datasets and network widths, although no label information is available during DBN training. Control experiments show that this effect depends on the learned feature structure and cannot be explained by random transformations, weight marginals, dimensionality reduction, or sigmoid saturation. The first hidden layers also frequently make class identity more accessible to linear and nonlinear probes. With greater depth, representations become increasingly compact and prototype-like as neurons acquire correlated feature directions. At the same time, GDV and probe accuracy reveal complementary aspects of class structure: improved average clustering can coexist with reduced accessibility for a few difficult class pairs. These findings demonstrate that layer-wise generative learning can spontaneously uncover and progressively amplify class-related structure in unlabeled data.

Problem

• Convergent Evolution
• Neural Representation
• Deep Belief Networks
• Unsupervised Learning
• Class Structure

Innovation

• Convergent Evolution
• Neural Representation
• Deep Belief Networks
• Unsupervised Learning
• Class Structure Emergence