Common-Mode Collapse and Recovery in Direct Feedback Alignment

2026-09-25

AI Summary

This study addresses the issue of training stagnation in Direct Feedback Alignment (DFA) caused by Tanh unit saturation induced by common-mode error components. Through mean-covariance decomposition, we elucidate the underlying common-mode error mechanism and establish a theoretical collapse model. To mitigate saturation, we propose a batch mean subtraction strategy. Combined with random projection analysis and comparisons against Adam, we validate its cross-architectural generalizability on MNIST and CIFAR-10. Our findings reveal that sign errors exacerbate collapse, whereas mean correction effectively restores performance. The proposed method preserves class-decoding capabilities while significantly reducing the learning time of the readout layer. Furthermore, it effectively alleviates training degradation in deep and convolutional networks, thereby accelerating convergence.

Abstract

Direct feedback alignment (DFA) trains hidden layers through fixed random projections of output error. With tanh hidden units and independent sigmoid outputs, plain stochastic gradient descent can stall near the loss of a constant predictor of class frequencies. We trace this stall to the error's common mode, the component shared across inputs. An exact mean-covariance decomposition separates a rank-one update formed by the mean teaching signal and mean presynaptic activity. Its leading component drives tanh units toward saturation. At initialization, random feedback provides no systematic correction of the shared error on average; readout learning limits its duration. A reduced model initialized from the network, without fitted parameters, predicts the concentration of activation sensitivity across 48 settings. On MNIST, class decodability largely survives collapse, but readout learning remains slow at a fixed learning rate. Adam learns faster despite deeper collapse. Calibrating the baseline readout to the class prior suppresses collapse and speeds learning; weaker feedback trades less collapse for slower learning. Replacing errors by their signs sustains collapse; subtracting the signal's batch mean prevents sustained collapse and improves learning in the tested setting. Related effects occur in deeper and convolutional networks and on CIFAR-10, with severity and cost depending on the readout, optimizer and input statistics.

Problem

• Direct Feedback Alignment
• Common-Mode Collapse
• Training Stagnation
• Error Signal
• Neural Network Optimization

Innovation

• Direct Feedback Alignment
• Common-Mode Collapse
• Mean-Covariance Decomposition
• Readout Calibration
• Batch Mean Subtraction