Self-Contrastive Forward-Forward Algorithm

2024-09-17

arXiv.org

AI Summary

Forward-Forward (FF) suffers from unreliable negative sample generation, limiting its performance on standard unsupervised benchmarks. To address this, we propose Self-Contrastive Forward-Forward (SC-FF), a purely forward-pass, backpropagation-free training framework grounded in self-supervised contrastive learning. SC-FF is the first to embed contrastive learning into FF, enabling automatic, cross-dataset generation of positive and negative samples. It is also the first to extend FF to recurrent neural networks, supporting sequential modeling. Furthermore, we introduce inter-layer local objective optimization and forward-gradient approximation for efficient parameter updates. Evaluated on MNIST, CIFAR-10, STL-10, and Tiny ImageNet, SC-FF significantly outperforms existing unsupervised local learning methods—achieving both high accuracy and low inference latency—thereby meeting the stringent requirements of lifelong learning on edge devices.

Technology Category

• Machine Learning: Unsupervised & Self-Supervised Learning
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

Agents that operate autonomously benefit from lifelong learning capabilities. However, compatible training algorithms must comply with the decentralized nature of these systems, which imposes constraints on both the parameter counts and the computational resources. The Forward-Forward (FF) algorithm is one of these. FF relies only on feedforward operations, the same used for inference, for optimizing layer-wise objectives. This purely forward approach eliminates the need for transpose operations required in traditional backpropagation. Despite its potential, FF has failed to reach state-of-the-art performance on most standard benchmark tasks, in part due to unreliable negative data generation methods for unsupervised learning. In this work, we propose the Self-Contrastive Forward-Forward (SCFF) algorithm, a competitive training method aimed at closing this performance gap. Inspired by standard self-supervised contrastive learning for vision tasks, SCFF generates positive and negative inputs applicable across various datasets. The method demonstrates superior performance compared to existing unsupervised local learning algorithms on several benchmark datasets, including MNIST, CIFAR-10, STL-10, and Tiny ImageNet. We extend FF's application to training recurrent neural networks, expanding its utility to sequential data tasks. These findings pave the way for high-accuracy, real-time learning on resource-constrained edge devices.

Problem

• Improves unsupervised learning in Forward-Forward algorithm
• Enhances performance on benchmark datasets like MNIST
• Extends algorithm to recurrent neural networks

Innovation

• Self-Contrastive Forward-Forward algorithm for local learning
• Generates positive and negative inputs contrastively
• Extends Forward-Forward to recurrent neural networks