neural network architecture

Existing neural networks are constrained by hierarchical tree-like architectures, which preclude direct communication among sibling nodes and prohibit backward signal propagation to higher-level modules—resulting in weak inter-module collaboration and inefficient representation learning. To address these limitations, we propose the Synchronous Graph Neural Architecture (SGNA), organizing neural units into a modular, dynamically collaborative graph structure that enables arbitrary node-to-node communication and cross-layer signal transmission. Our key contributions are threefold: (1) introducing the first modular graph-structured paradigm for neural architecture design; (2) developing a systematic regularization framework to enforce module independence and load balancing; and (3) generalizing neural architecture search (NAS) to the space of directed acyclic graphs (DAGs). Extensive multi-task experiments demonstrate that SGNA significantly outperforms deep stacked baselines under parameter constraints, achieving superior collaborative representation capability and more comprehensive coverage of the search space.

Non-modular neural networks suffer from exponential sample complexity growth with input dimensionality in high-dimensional combinatorial tasks—a fundamental bottleneck for generalization. Method: We theoretically and empirically investigate modular neural networks’ generalization mechanisms. We first establish, for the first time, a rigorous theoretical guarantee that modular architectures achieve dimension-independent sample complexity. Methodologically, we propose a task-intrinsic dimensionality–driven modular architecture design and a theory-guided learning rule. Results: Experiments demonstrate significant improvements over baselines in both in-distribution and out-of-distribution generalization, achieving dimension-agnostic efficient learning on high-dimensional combinatorial tasks—thereby breaking conventional scaling laws. Our core contributions are threefold: (i) establishing the first formal theoretical foundation for modular generalization; (ii) devising a provably optimal, theory-grounded learning mechanism; and (iii) empirically validating its fundamental superiority for high-dimensional combinatorial generalization.

Addressing the fundamental limitations of artificial neural networks—namely, poor systematic generalization, catastrophic forgetting, few-shot learning inefficiency, and inadequate multi-step reasoning due to the absence of human-like cognitive development mechanisms—this paper proposes a meta-learning framework explicitly optimizing for “motivation + practice.” Unlike conventional paradigms driven by indirect objectives (e.g., loss minimization), our framework integrates differentiable optimization, curriculum learning, task embedding, and practice trajectory modeling to enable models to autonomously acquire skill-improvement motivation and structured training opportunities during learning. Evaluated across four benchmark task families, it significantly outperforms state-of-the-art methods, demonstrating the efficacy of the motivation-practice mechanism for robust generalization and continual learning. Moreover, it establishes, for the first time, a computationally tractable cognitive development pathway for neural networks—bridging machine learning and cognitive science through a novel, principled paradigm.

This work investigates how neural network width governs training dynamics. For single-hidden-layer linear networks, we derive the first exact analytical solution of learning dynamics at arbitrary finite width, unifying the characterization of the two-phase evolution—kernel learning and feature learning—and establishing a complete phase diagram parameterized by width, layer-wise learning rates, and initialization scale. Methodologically, we integrate analytical dynamical systems analysis, phase-diagram modeling, and empirical validation on nonlinear networks. Crucially, we identify three novel mechanisms operative during the feature-learning phase: alignment learning, de-alignment learning, and rescaling learning—each transcending the conventional kernel-method paradigm. These theoretical insights are empirically reproduced in realistic deep networks, offering a new conceptual framework for understanding training dynamics and designing adaptive optimization algorithms. (138 words)

This work addresses the fundamental trade-off between standard accuracy and adversarial robustness in supervised learning. Methodologically, it introduces the first architecture-level accuracy–robustness trade-off curve, quantifying the inverse relationship between these objectives across diverse neural network architectures; defines a sensitivity influence function to theoretically characterize the stability of optimal solutions under adversarial perturbations; and reveals—via theoretical analysis of overparameterized linear models—that adversarial training implicitly regularizes model dynamics, interpolating between L₁ (LASSO) and L₂ (ridge regression) behaviors. The approach integrates rigorous theoretical analysis, influence-function-based modeling, and extensive empirical evaluation across fully connected, deep, and varying-width networks. Results consistently validate the existence and structure of the trade-off, providing an interpretable, predictive theoretical foundation for principled neural architecture selection.

This study investigates how network architecture influences the stability-plasticity trade-off and the interplay between interference and transfer in continual learning. By systematically comparing modular and monolithic recurrent networks under controlled task similarity and weight initialization scales—and integrating effective dimensionality analysis—the work identifies representational dimensionality as a critical factor determining the efficacy of architectural separation. The findings reveal that in low-dimensional (representationally rich) regimes, modular networks substantially outperform baselines by adaptively shaping a hierarchical representational geometry aligned with task similarity, thereby achieving alignment and orthogonality of task-specific subspaces. In contrast, architectural differences exhibit negligible effects in high-dimensional regimes.

This work investigates whether the grokking phenomenon observed in Transformers on modular addition tasks stems from architectural degrees of freedom and proposes structural interventions to eliminate the memorization phase. By introducing spherical embeddings (L2-normalized residual streams), a fixed-temperature decoder, and uniform attention mechanisms, the study imposes geometric inductive biases to actively shape training dynamics. Experiments demonstrate that this approach reduces the grokking onset time by over 20-fold on modular addition while achieving 100% generalization; however, no acceleration is observed on the S5 task, highlighting the critical role of alignment between task symmetry and inductive bias. This study presents the first causal evidence—through targeted architectural intervention—that architectural freedom underlies grokking.

This work addresses the diminished understanding of neural network fundamentals caused by the widespread use of high-level deep learning libraries. To bridge this gap, the authors construct a complete neural network framework from scratch, eschewing automatic differentiation and prebuilt modules. The implementation explicitly details forward and backward propagation, incorporates multiple activation functions, L2 regularization, and advanced optimizers such as Adam. Designed to balance pedagogical clarity with engineering scalability, the framework demonstrates numerical stability, correctness, and generalization capability on multiclass classification tasks. It thus provides a reproducible and extensible tool for both research and instruction, fostering deeper insight into the core principles of deep learning.

This study investigates the mechanisms underlying cognitive flexibility in dynamic multitask environments—specifically, how agents can effectively generalize to novel tasks while preserving the stability of previously acquired knowledge. To this end, the authors construct a multitask learning environment based on dual cue dimensions and introduce, for the first time, graph-theoretic measures to characterize the connectivity structure among tasks. They propose an attention-based model integrating gating and concatenation mechanisms that decomposes task structure and allocates attention sequentially, enabling a systematic analysis of how environmental structure and model architecture jointly shape cognitive flexibility. Experimental results demonstrate that environmental richness and task connectivity significantly modulate the trade-off between stability and generalization, with the proposed attention model exhibiting particularly pronounced advantages in highly connected task environments.

Existing deep learning models struggle to effectively encode spatial, topological, and semantic structural information inherent in images. This work systematically evaluates the impact of various visual graph construction strategies on image classification performance within a unified three-layer Graph Convolutional Network (GCN) framework. For the first time, it demonstrates that the graph structure itself plays a decisive role in model performance. The study underscores the critical importance of the graph construction preprocessing stage, providing empirical evidence that well-designed graph structures substantially enhance classification accuracy. These findings offer both methodological guidance and practical justification for graph structure selection and preprocessing in visual graph neural networks.