The Price of Locality: Why Forward-Forward Underperforms Backpropagation?

2026-09-27

AI Summary

This study addresses the significant performance degradation of the Forward-Forward Algorithm (FFA) relative to backpropagation (BP) as network depth increases, a phenomenon whose theoretical origins remain unclear. To diagnose the failure mechanisms of FFA, this work integrates Polyak-Łojasiewicz inequalities, inter-layer distribution shift modeling, and representation geometric collapse theory. We provide the first rigorous proof that the representational diversity of FFA decays exponentially with depth, revealing how concurrent updates impose optimization lower bounds and how geometric collapse constrains learning capacity. Furthermore, we elucidate the advantage of BP in preserving gradient diversity via the chain rule. This research establishes a solid theoretical foundation for understanding the performance bottlenecks inherent in FFA.

Abstract

The Forward-Forward Algorithm (FFA) replaces backpropagation (BP) with layer-wise local contrastive objectives, eliminating the backward pass and the need to retain intermediate activations, yet suffers a persistent performance gap with BP that worsens with depth. This paper diagnoses two structural failure modes: an optimization floor arising from concurrent local updates; and a geometric collapse of layer representations driven by the local update mechanism. On the optimization side, we prove that the FFA loss satisfies the Polyak--Lojasiewicz inequality at each layer; however, simultaneous layer updates induce inter-layer representation-distribution drift, so each layer optimizes against a moving input distribution and incurs an error floor. On the representational side, the pairwise similarity kernel of layer representations contracts exponentially toward rank one as depth increases, collapsing the diversity of per-layer error signals. This collapse bounds FFA's effective learning capacity, which measures the diversity of gradient information across layers, independently of depth, whereas BP's chain-rule signal preserves per-layer diversity, yielding a capacity that scales with depth.

Problem

• Forward-Forward Algorithm
• Backpropagation
• Local learning
• Representation collapse
• Optimization floor

Innovation

• Forward-Forward Algorithm
• Polyak-Lojasiewicz inequality
• representation-distribution drift
• geometric collapse
• effective learning capacity