On the Infinite Width and Depth Limits of Predictive Coding Networks

2026-02-07

AI Summary

This work investigates the scalability of predictive coding networks in the joint limit of infinite width and depth, and establishes their theoretical connection to backpropagation. By analyzing linear residual networks under this asymptotic regime—leveraging energy function optimization, the activity equilibrium assumption, and reparameterization techniques—the study demonstrates for the first time that predictive coding and backpropagation exhibit identical feature learning capabilities under a stable wide-and-deep parameterization. Furthermore, their gradients are provably equivalent in the regime of large width and small depth. Theoretical analysis shows that predictive coding converges to the backpropagation loss under specific conditions, and experiments confirm that this equivalence extends to nonlinear deep networks that satisfy the activity equilibrium assumption.

Technology Category

• Machine Learning: Deep Learning Theory
• Computer Vision: Learning & Optimization for CV
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Social Networks and Social Media: Influence propagation, information diffusion, and the prediction on networks
• Economics, Online Markets and Human Computation: Social networks and social learning
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Predictive coding (PC) is a biologically plausible alternative to standard backpropagation (BP) that minimises an energy function with respect to network activities before updating weights. Recent work has improved the training stability of deep PC networks (PCNs) by leveraging some BP-inspired reparameterisations. However, the full scalability and theoretical basis of these approaches remains unclear. To address this, we study the infinite width and depth limits of PCNs. For linear residual networks, we show that the set of width- and depth-stable feature-learning parameterisations for PC is exactly the same as for BP. Moreover, under any of these parameterisations, the PC energy with equilibrated activities converges to the BP loss in a regime where the model width is much larger than the depth, resulting in PC computing the same gradients as BP. Experiments show that these results hold in practice for deep nonlinear networks, as long as an activity equilibrium seem to be reached. Overall, this work unifies various previous theoretical and empirical results and has potentially important implications for the scaling of PCNs.

Problem

• predictive coding
• infinite width
• infinite depth
• scalability
• theoretical foundation

Innovation

• predictive coding
• infinite width limit
• infinite depth limit
• feature learning
• gradient equivalence