🤖 AI Summary
This work investigates the dynamic evolution of layer-wise and global accuracy during Forward-Forward (FF) algorithm training. Addressing three core questions—(i) layer-wise accuracy dynamics, (ii) the impact of depth on convergence speed, and (iii) the correlation between per-layer accuracy and overall model performance—we propose an analytical framework based on dual forward passes and layer-local loss functions, enabling fine-grained accuracy tracking and correlation modeling across layers. We systematically uncover a “deep-layer lag” phenomenon in FF networks: shallow layers attain high accuracy significantly earlier than deep layers, and their early accuracy strongly predicts final model performance (Pearson *r* > 0.95). This provides a mechanistic explanation for effective backpropagation-free learning. Our findings empirically validate the hierarchical, cooperative nature of FF training at the dynamical level, advancing theory for brain-inspired efficient learning.
📝 Abstract
The Forward-Forward algorithm is an alternative learning method which consists of two forward passes rather than a forward and backward pass employed by backpropagation. Forward-Forward networks employ layer local loss functions which are optimized based on the layer activation for each forward pass rather than a single global objective function. This work explores the dynamics of model and layer accuracy changes in Forward-Forward networks as training progresses in pursuit of a mechanistic understanding of their internal behavior. Treatments to various system characteristics are applied to investigate changes in layer and overall model accuracy as training progresses, how accuracy is impacted by layer depth, and how strongly individual layer accuracy is correlated with overall model accuracy. The empirical results presented suggest that layers deeper within Forward-Forward networks experience a delay in accuracy improvement relative to shallower layers and that shallower layer accuracy is strongly correlated with overall model accuracy.