Expressivity of Shallow Neural Networks Over Finite Fields

2026-07-19

AI Summary

This work investigates the expressive power of shallow polynomial neural networks over finite fields and its dependence on the field characteristic. By mapping network weights into a product of polynomial rings, we define a neural manifold and quantify expressivity via its cardinality. Leveraging tools from algebraic geometry—particularly rational point counting and the Weil conjectures—we analyze the structure of this manifold and establish tight upper and lower bounds on its size. Crucially, we demonstrate for the first time that the field characteristic fundamentally governs expressive capacity: we construct explicit network architectures whose behavior diverges markedly between fields of characteristic zero and positive characteristic, thereby revealing the pivotal role of finite field characteristics in determining neural network expressivity.

Abstract

We study the expressivity of shallow polynomial neural networks (PNNs) with monomial activation functions over finite fields. For a given architecture, we define a neuromanifold as the image of the map from all possible network weights into the product of polynomial rings. We quantify the expressivity by the cardinality of the neuromanifold, and derive a natural lower and upper bound. This leads to counting rational points over finite fields, a problem closely linked to the Weil conjectures. Finally, we present an architecture that exhibits a striking difference in the neuromanifolds when considered over a characteristic zero versus a finite-characteristic field, illustrating the critical role of field characteristic in the notion of expressivity.

Problem

• expressivity
• shallow neural networks
• finite fields
• polynomial neural networks
• field characteristic

Innovation

• neuromanifold
• polynomial neural networks
• finite fields
• expressivity
• Weil conjectures