Scalable Forward-Forward Algorithm

2025-01-06

AI Summary

To address the high computational cost, substantial memory footprint, and poor modularity of backpropagation (BP), this paper proposes a scalable Forward-Forward (FF) training framework. Methodologically, it introduces three key innovations: (1) the first adaptation of FF to modern convolutional architectures (e.g., MobileNetV3, ResNet18), featuring a convolutional-feature-space-specific loss function; (2) a hybrid “in-block backpropagation + inter-block FF” paradigm that balances training stability and modular decomposition; and (3) integration of block-wise training, few-shot fine-tuning, and transfer learning strategies. Experiments on standard image classification benchmarks demonstrate that pure FF achieves accuracy comparable to BP; the hybrid paradigm surpasses the BP baseline in accuracy on ResNet18 while maintaining comparable training speed; and the approach exhibits显著 robustness and adaptability under data-scarce and cross-domain transfer scenarios.

Technology Category

• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization

Abstract

We propose a scalable Forward-Forward (FF) algorithm that eliminates the need for backpropagation by training each layer separately. Unlike backpropagation, FF avoids backward gradients and can be more modular and memory efficient, making it appealing for large networks. We extend FF to modern convolutional architectures, such as MobileNetV3 and ResNet18, by introducing a new way to compute losses for convolutional layers. Experiments show that our method achieves performance comparable to standard backpropagation. Furthermore, when we divide the network into blocks, such as the residual blocks in ResNet, and apply backpropagation only within each block, but not across blocks, our hybrid design tends to outperform backpropagation baselines while maintaining a similar training speed. Finally, we present experiments on small datasets and transfer learning that confirm the adaptability of our method.

Problem

• Forward-Forward Algorithm
• Neural Network Training
• Resource Efficiency

Innovation

• Forward-Forward Algorithm
• Self-learning Layers
• Local Backpropagation Integration