direct feedback alignment

This work addresses the scalability limitations of Feedback Alignment (FA) in deep networks, which stem from the excessively low effective rank of error signals. The study identifies low-dimensional gradient dynamics as a critical bottleneck underlying FA’s failure to scale effectively. To enhance the representational capacity of error signals, the authors propose increasing the geometric dimensionality of both weight updates and activations: they employ the Muon optimizer to orthogonalize weight updates and introduce activation normalization in hidden layers to promote activation orthogonality. Evaluated on CIFAR-100, this approach improves the test accuracy of ResNet-18 by 9 percentage points over standard FA, substantially advancing the training performance of deep networks under the FA framework.

A key biological implausibility of backpropagation lies in its requirement for precise weight symmetry between forward and backward pathways. To address this, we propose Product Feedback Alignment (PFA), a novel learning algorithm that replaces the fixed random feedback matrix in classical Feedback Alignment with a dynamic, multiplicative feedback mechanism. PFA is the first method—both theoretically and empirically—to achieve high-fidelity approximation of standard backpropagation gradients while fully eliminating the weight symmetry constraint. The algorithm is natively compatible with convolutional neural networks (CNNs) and mainstream optimizers. On standard image recognition benchmarks—including ImageNet—it matches backpropagation’s accuracy, substantially outperforms classical Feedback Alignment, and exhibits superior convergence stability and generalization performance in deep architectures. By reconciling gradient-based learning with biologically plausible circuitry, PFA advances the frontier of biologically interpretable deep learning.

Inspired by developmental neuroscience, this work investigates whether spontaneous stochastic neural activity can enhance learning efficiency and generalization in artificial neural networks. To this end, we propose a weight-transfer-free pretraining framework that injects structured random noise into network dynamics and integrates feedback alignment (FA) to enable self-organized alignment between forward and backward pathways. Theoretically and empirically, we demonstrate that this pretraining spontaneously induces low-rank weight structure, reduces effective dimensionality, and biases optimization toward simpler solutions. It significantly accelerates convergence—matching backpropagation in speed—while reducing generalization error, improving out-of-distribution (OOD) robustness, enhancing meta-loss optimization, and facilitating multi-task adaptation. Our key contribution is the first identification and exploitation of intrinsic weight alignment and implicit regularization emerging from random-noise-driven dynamics, establishing a biologically plausible, efficient, and robust training paradigm for neural networks.

Traditional neural network training relies on fixed optimization pipelines, rendering it inflexible in dynamically addressing training instability and anomalies. To address this limitation, we propose the first interactive training framework enabling real-time human–AI collaborative intervention. Our method employs a lightweight control server that integrates expert human directives with AI agent feedback to dynamically adjust hyperparameters, data sampling strategies, and model checkpoints during training. This framework introduces, for the first time, a closed-loop interactive paradigm into neural network training, establishing a scalable human–machine collaboration interface coupled with automated response mechanisms. Experimental results demonstrate significant improvements in training stability, reduced sensitivity to initial hyperparameter configurations, and enhanced real-time responsiveness to user-specified customization requirements. The effectiveness is validated across multiple benchmark tasks.

Addressing the fundamental limitations of artificial neural networks—namely, poor systematic generalization, catastrophic forgetting, few-shot learning inefficiency, and inadequate multi-step reasoning due to the absence of human-like cognitive development mechanisms—this paper proposes a meta-learning framework explicitly optimizing for “motivation + practice.” Unlike conventional paradigms driven by indirect objectives (e.g., loss minimization), our framework integrates differentiable optimization, curriculum learning, task embedding, and practice trajectory modeling to enable models to autonomously acquire skill-improvement motivation and structured training opportunities during learning. Evaluated across four benchmark task families, it significantly outperforms state-of-the-art methods, demonstrating the efficacy of the motivation-practice mechanism for robust generalization and continual learning. Moreover, it establishes, for the first time, a computationally tractable cognitive development pathway for neural networks—bridging machine learning and cognitive science through a novel, principled paradigm.

This study systematically evaluates five learning algorithms—including feedback alignment (FA), its variants, and standard backpropagation (BP)—within a unified convolutional architecture on CIFAR-10, examining their biological plausibility, representational interpretability, and computational complexity. Although FA is biologically plausible, it has proven difficult to scale effectively in convolutional networks without compromising its neuroscientific foundations. The findings reveal that improved FA variants achieve efficient training by approximating the internal representational geometry of BP, converging to functionally equivalent representations despite differing weight-update mechanisms. This demonstrates that the key to FA’s success lies in its ability to effectively approximate BP’s representational space, thereby reconciling biological plausibility with high performance.

This work investigates whether backpropagation is sample-efficiently optimal and proposes synthetic gradients as a viable alternative. By constructing a unified vectorized feedback framework that jointly models loss-based and reward-based learning within a single computational graph, the study establishes, for the first time, sufficient theoretical conditions under which synthetic gradients provably surpass backpropagation in sample efficiency—advantages that can be made arbitrarily large. The analysis centers on the mean squared error properties of gradient estimators and demonstrates, through contextual bandit and reinforcement learning tasks, the substantial potential of synthetic gradients to enhance sample efficiency.

This work addresses critical limitations in the evaluation of existing feedback alignment methods, which rely primarily on task accuracy and aggregated gradient cosine similarity and thus fail to uncover two silent failure modes: reference gradient degradation and inter-layer heterogeneity in credit assignment. To resolve this, the authors propose a novel diagnostic evaluation protocol that introduces three layer-wise checks—scale stability, reference validity, and depth utility—combined with fine-grained cosine analysis. This framework enables, for the first time, the explicit identification and disentanglement of the aforementioned failure mechanisms. Empirical validation across diverse architectures and algorithms demonstrates its ability to detect failure cases missed by conventional metrics and to provide actionable insights for improvement, confirming both its broad applicability and high diagnostic precision.

This work addresses the training instability in Direct Feedback Alignment (DFA) caused by anisotropy in either presynaptic activities or local error signals. To mitigate this issue, the authors propose a normalized DFA approach that conditionally normalizes both activities and error signals. They introduce a symmetric, block-wise dual-factor normalization framework, decoupling and empirically validating the independent contributions of activity and error conditioning for the first time. The method is theoretically supported by linearized spectral analysis and implemented via inverse second-moment preconditioning and Kronecker-factor approximations. Controlled experiments on MNIST and Fashion-MNIST demonstrate that activity conditioning alone yields performance gains of up to 40 percentage points, while error conditioning improves accuracy by 1.77–7.53 percentage points; combining both strategies provides further modest improvements.

本文探讨了神经计算中前向-后向不匹配问题，通过分析状态动态、信用分配和生物学基础的关系，提出了一种分类方法以解决该问题。