Pretraining with Random Noise for Fast and Robust Learning without Weight Transport

2024-05-27

Neural Information Processing Systems

AI Summary

Inspired by developmental neuroscience, this work investigates whether spontaneous stochastic neural activity can enhance learning efficiency and generalization in artificial neural networks. To this end, we propose a weight-transfer-free pretraining framework that injects structured random noise into network dynamics and integrates feedback alignment (FA) to enable self-organized alignment between forward and backward pathways. Theoretically and empirically, we demonstrate that this pretraining spontaneously induces low-rank weight structure, reduces effective dimensionality, and biases optimization toward simpler solutions. It significantly accelerates convergence—matching backpropagation in speed—while reducing generalization error, improving out-of-distribution (OOD) robustness, enhancing meta-loss optimization, and facilitating multi-task adaptation. Our key contribution is the first identification and exploitation of intrinsic weight alignment and implicit regularization emerging from random-noise-driven dynamics, establishing a biologically plausible, efficient, and robust training paradigm for neural networks.

Technology Category

Machine Learning: Deep Neural Architectures and Foundation ModelsNatural Language Processing: Learning & Optimization for NLPSearch and Optimization: Learning to Search

Application Category

Web Mining and Content Analysis: Large pretrained models with web dataSearch and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for rankingGraph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs

Abstract

The brain prepares for learning even before interacting with the environment, by refining and optimizing its structures through spontaneous neural activity that resembles random noise. However, the mechanism of such a process has yet to be thoroughly understood, and it is unclear whether this process can benefit the algorithm of machine learning. Here, we study this issue using a neural network with a feedback alignment algorithm, demonstrating that pretraining neural networks with random noise increases the learning efficiency as well as generalization abilities without weight transport. First, we found that random noise training modifies forward weights to match backward synaptic feedback, which is necessary for teaching errors by feedback alignment. As a result, a network with pre-aligned weights learns notably faster than a network without random noise training, even reaching a convergence speed comparable to that of a backpropagation algorithm. Sequential training with both random noise and data brings weights closer to synaptic feedback than training solely with data, enabling more precise credit assignment and faster learning. We also found that each readout probability approaches the chance level and that the effective dimensionality of weights decreases in a network pretrained with random noise. This pre-regularization allows the network to learn simple solutions of a low rank, reducing the generalization loss during subsequent training. This also enables the network robustly to generalize a novel, out-of-distribution dataset. Lastly, we confirmed that random noise pretraining reduces the amount of meta-loss, enhancing the network ability to adapt to various tasks. Overall, our results suggest that random noise training with feedback alignment offers a straightforward yet effective method of pretraining that facilitates quick and reliable learning without weight transport.

Problem

• Understanding how random noise pretraining enhances learning efficiency
• Exploring feedback alignment's role in neural network pretraining
• Investigating generalization improvement via random noise pre-regularization

Innovation

• Pretraining neural networks with random noise
• Using feedback alignment for error teaching
• Reducing generalization loss via pre-regularization