Spectral Higher-Order Neural Networks Have Sharp Expressivity Bounds

2026-07-21

AI Summary

This work addresses the challenge of deploying high-order neural networks, whose parameter counts grow exponentially with order due to hyperedge complexity. To mitigate this, the authors propose a spectral graph theory–based parameterization scheme that leverages spectral features to enable weight sharing, drastically reducing model parameters while preserving expressive power and architectural flexibility. Evaluated on the challenging N-bit parity benchmark—a task known for its sensitivity to model expressivity—the proposed architecture achieves superior performance with significantly fewer parameters and demonstrates enhanced interpretability. These results empirically validate the theoretical advantages of the approach in balancing efficiency and representational capacity.

Abstract

Neural hypergraphs are a natural generalization of neural networks, the reference models in modern machine learning. Yet, their deployment has proven demanding: the number of weighted hyperedges required leads to an intractable parameter explosion. However, a novel parametrization that leverages spectral attributes for neural hypergraphs has been recently proposed, that enables to recycle parameters via a weight sharing scheme and consequently yields a significant reduction of the associated computational cost. Preliminary tests carried out on spectral higher-order architectures pointed to meaningful improvements in both performance and interpretability. Building on these results, we advance the benchmarking efforts by evaluating the spectral higher order framework on N-bit parity tasks, a well-established testbed known to be particularly challenging. As we will convincingly argue, Spectral Higher-Order Neural Networks (SHONNs) possess a versatile and highly tunable hypothesis space.

Problem

• neural hypergraphs
• parameter explosion
• computational cost
• higher-order neural networks

Innovation

• spectral parametrization
• neural hypergraphs
• weight sharing
• higher-order neural networks
• expressivity bounds