Sparse Goodness: How Selective Measurement Transforms Forward-Forward Learning

2026-03-28

AI Summary

This work addresses a key limitation of the traditional Forward-Forward (FF) algorithm, which relies on the sum of squared activations as its goodness function and consequently fails to effectively capture the discriminative structure of neural activity. The study is the first to argue that a goodness function should prioritize the shape of neural responses over their total energy. To this end, it introduces two novel classes of scale-invariant goodness functions: selectivity-based functions (e.g., top-k and entmax-weighted energy) and shape-sensitive functions (e.g., kurtosis and higher-order moments). When integrated with decoupled label-feature forward propagation (FFCL), the proposed approach achieves 98.2% accuracy on MNIST and 89.0% on Fashion-MNIST—improving up to 32.6 percentage points over baseline methods—and consistently outperforms conventional FF across additional benchmarks including USPS and SVHN.

Technology Category

• Machine Learning: Active Learning
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Search and Optimization: Learning to Search

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

The Forward-Forward (FF) algorithm is a biologically plausible alternative to backpropagation that trains neural networks layer by layer using a local goodness function to distinguish positive from negative data. Since its introduction, sum-of-squares (SoS) has served as the default goodness function. In this work, we systematically study the design space of goodness functions, investigating both which activations to measure and how to aggregate them. We introduce top-k goodness, which evaluates only the k most active neurons, and show that it substantially outperforms SoS, improving Fashion-MNIST accuracy by 22.6 percentage points. We further introduce entmax-weighted energy, which replaces hard top-k selection with a learnable sparse weighting based on the alpha-entmax transformation, yielding additional gains. Orthogonally, we adopt separate label feature forwarding (FFCL), in which class hypotheses are injected at every layer through a dedicated projection rather than concatenated only at the input. Combining these ideas, we achieve 87.1 percent accuracy on Fashion-MNIST with a 4x2000 architecture, representing a 30.7 percentage point improvement over the SoS baseline while changing only the goodness function and the label pathway. Across controlled experiments covering 11 goodness functions, two architectures, and a sparsity spectrum analysis over both k and alpha, we identify a consistent principle: sparsity in the goodness function is the most important design choice in FF networks. In particular, adaptive sparsity with alpha approximately 1.5 outperforms both fully dense and fully sparse alternatives.

Problem

• Forward-Forward
• goodness function
• selectivity
• shape sensitivity
• heavy-tailed distributions

Innovation

• Forward-Forward algorithm
• goodness function
• heavy-tailed activation
• burstiness
• scale-invariant statistics