Technology Category
Machine Learning: Deep Neural Architectures and Foundation ModelsNatural Language Processing: Learning & Optimization for NLPSearch and Optimization: Learning to Search
Training Large Neural Networks With Low-Dimensional Error Feedback
🤖 AI Summary
Standard backpropagation in deep neural networks is computationally expensive and biologically implausible due to its requirement for high-dimensional, symmetric weight updates.
Method: This work proposes a feedback-alignment-based local learning framework that decouples forward and feedback pathways, enabling efficient error-driven learning via low-dimensional error signals—e.g., projecting gradients onto task-relevant subspaces of dimensionality as low as 10. The framework supports linear, convolutional, and Transformer architectures and incorporates theoretically grounded nonlinear expansion rules.
Contribution/Results: It achieves, for the first time, efficient convergence under feedback alignment in convolutional networks on ImageNet-scale tasks, matching standard backpropagation accuracy. A theoretical convergence guarantee is established for the linear case, and empirical results demonstrate significantly reduced training error. This work establishes a new paradigm for scalable, biologically plausible, and computationally lightweight learning in large models.
Technology Category
Application Category
📝 Abstract
Training deep neural networks typically relies on backpropagating high dimensional error signals a computationally intensive process with little evidence supporting its implementation in the brain. However, since most tasks involve low-dimensional outputs, we propose that low-dimensional error signals may suffice for effective learning. To test this hypothesis, we introduce a novel local learning rule based on Feedback Alignment that leverages indirect, low-dimensional error feedback to train large networks. Our method decouples the backward pass from the forward pass, enabling precise control over error signal dimensionality while maintaining high-dimensional representations. We begin with a detailed theoretical derivation for linear networks, which forms the foundation of our learning framework, and extend our approach to nonlinear, convolutional, and transformer architectures. Remarkably, we demonstrate that even minimal error dimensionality on the order of the task dimensionality can achieve performance matching that of traditional backpropagation. Furthermore, our rule enables efficient training of convolutional networks, which have previously been resistant to Feedback Alignment methods, with minimal error. This breakthrough not only paves the way toward more biologically accurate models of learning but also challenges the conventional reliance on high-dimensional gradient signals in neural network training. Our findings suggest that low-dimensional error signals can be as effective as high-dimensional ones, prompting a reevaluation of gradient-based learning in high-dimensional systems. Ultimately, our work offers a fresh perspective on neural network optimization and contributes to understanding learning mechanisms in both artificial and biological systems.
Problem
Research questions and friction points this paper is trying to address.
Proposes low-dimensional error signals for training large neural networks.
Introduces a novel local learning rule using Feedback Alignment.
Challenges reliance on high-dimensional gradients in neural network training.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Low-dimensional error feedback for training
Decouples forward and backward network passes
Efficient training of convolutional networks
🔎 Similar Papers
No similar papers found.
