Heterogeneous Connectivity in Sparse Networks: Fan-in Profiles, Gradient Hierarchy, and Topological Equilibria

2026-04-12

AI Summary

This study investigates the impact of heterogeneous connection structures on model performance in sparse neural networks, with a focus on the role of hub neuron configurations. By constructing static sparse networks with parameterized fan-in distributions—such as log-normal and power-law—and integrating them with RigL dynamic sparse training alongside gradient concentration analysis, the authors evaluate performance across multiple datasets and sparsity levels. The findings reveal that randomly placed hub neurons yield no significant benefit, whereas task-aligned, optimization-driven hub placement enhances performance. Moreover, RigL training tends to converge toward balanced fan-in distributions, and initializing networks accordingly accelerates convergence. At 90% sparsity, this strategy improves accuracy by 0.16%, 0.43%, and 0.49% on Fashion-MNIST, EMNIST, and Forest Cover, respectively.

Technology Category

Machine Learning: Mixture of Experts (MoE)Cognitive Modeling & Cognitive Systems: Neural Spike CodingSearch and Optimization: Non-convex Optimization

Application Category

Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphsResponsible Web: Human-perceived consequences of algorithmic deployment on the webWeb Mining and Content Analysis: Large pretrained models with web data

Abstract

Profiled Sparse Networks (PSN) replace uniform connectivity with deterministic, heterogeneous fan-in profiles defined by continuous, nonlinear functions, creating neurons with both dense and sparse receptive fields. We benchmark PSN across four classification datasets spanning vision and tabular domains, input dimensions from 54 to 784, and network depths of 2--3 hidden layers. At 90% sparsity, all static profiles, including the uniform random baseline, achieve accuracy within 0.2-0.6% of dense baselines on every dataset, demonstrating that heterogeneous connectivity provides no accuracy advantage when hub placement is arbitrary rather than task-aligned. This result holds across sparsity levels (80-99.9%), profile shapes (eight parametric families, lognormal, and power-law), and fan-in coefficients of variation from 0 to 2.5. Internal gradient analysis reveals that structured profiles create a 2-5x gradient concentration at hub neurons compared to the ~1x uniform distribution in random baselines, with the hierarchy strength predicted by fan-in coefficient of variation ($r = 0.93$). When PSN fan-in distributions are used to initialise RigL dynamic sparse training, lognormal profiles matched to the equilibrium fan-in distribution consistently outperform standard ERK initialisation, with advantages growing on harder tasks, achieving +0.16% on Fashion-MNIST ($p = 0.036$, $d = 1.07$), +0.43% on EMNIST, and +0.49% on Forest Cover. RigL converges to a characteristic fan-in distribution regardless of initialisation. Starting at this equilibrium allows the optimiser to refine weights rather than rearrange topology. Which neurons become hubs matters more than the degree of connectivity variance, i.e., random hub placement provides no advantage, while optimisation-driven placement does.

Problem

• heterogeneous connectivity
• sparse networks
• fan-in profiles
• gradient hierarchy
• topological equilibria

Innovation

• Profiled Sparse Networks
• heterogeneous connectivity
• fan-in profiles
• dynamic sparse training
• topological equilibria