activation function design

This work addresses the lack of a unified formulation among conventional activation functions and their hardware deployment limitations imposed by ADC/DAC bottlenecks. The authors propose Threshold Gating (TG) as a universal primitive for neural nonlinearities, employing an input-conditioned branch gating mechanism that subsumes major activation functions as special cases. Building on this unification, they introduce the “Minimal Branch Theorem,” enabling lossless activation function conversion and seamless model transfer across diverse architectures—such as CNNs, Transformers, and RNNs—without retraining. The framework supports both training from scratch and post-hoc adaptation, offering benefits in model compression, performance enhancement, and training acceleration. Furthermore, it maps efficiently onto analog in-memory computing architectures, substantially reducing power consumption and area overhead.

This work addresses the limited expressiveness of fixed activation functions in neural networks and the substantial computational overhead introduced by existing learnable activations. We propose a lightweight, general-purpose parametric nonlinearity enhancement framework that dynamically modulates the nonlinear behavior of standard activation functions (e.g., ReLU) via an end-to-end differentiable, adaptive parameterization. This enables structural adaptability while improving model performance. The framework is plug-and-play, fully compatible with standard CNN architectures, and introduces only a negligible number of learnable parameters. Extensive experiments on MNIST and multiple CNN benchmark tasks demonstrate significant improvements in classification accuracy, with near-negligible increases in computational cost (<0.5% additional FLOPs). Empirical results validate its strong generalization capability and practical utility.

High inter-layer communication overhead severely limits the energy efficiency of artificial neural networks. Method: Inspired by nonlinear dendritic computation in biological neurons, we propose a differentiable dendritic computing unit that embeds local feature aggregation within individual neurons. Our approach integrates computational neuroscience modeling with end-to-end machine learning training to construct a dendrite-enhanced network architecture supporting gradient backpropagation. Contribution/Results: While dendritic nonlinearity alone does not enhance baseline learning capability, performing nonlinear aggregation of multi-branch inputs *within* each neuron significantly expands representational capacity and reduces inter-layer data transmission by up to 62% (on ImageNet using ResNet-18), without compromising inference accuracy. This work establishes a “communication-minimization” paradigm for neuromorphic accelerator design targeting low-power edge AI.

How do structural features of connectivity—such as rapidly decaying singular value spectra and structured singular vector overlaps—regulate high-dimensional collective dynamics in nonlinear recurrent neural networks? Method: We introduce a random modal model that unifies the effects of input/output mode overlap, single-neuron heterogeneity, and low-rank connectivity structure. Employing path-integral saddle-point analysis, two-node cavity methods, and random matrix theory, we derive analytical expressions for both the dimensionality of neural activity and its temporal correlation spectrum. Contribution/Results: We establish, for the first time, a quantitative relationship between the effective rank of the coupling matrix and the intrinsic dimensionality of neural activity. Crucially, we find that low-dimensional connectivity manifests exclusively at the population level—individual neuron statistics remain indistinguishable from high-dimensional controls. Our framework yields testable dynamical predictions for connectomes reconstructed via electron microscopy (EM), bridging structural anatomy and functional dynamics.

This work challenges the conventional infinite-width assumption by investigating whether pre-activation distributions in finite-width neural networks can strictly retain Gaussianity. We derive the first necessary and sufficient constraints ensuring Gaussian pre-activations throughout training, thereby reformulating edge-of-chaos theory with precise analytical characterization. Our framework unifies diverse initialization schemes and systematically evaluates Gaussianity as an optimality criterion for initialization. Leveraging probabilistic propagation modeling, constraints on nonlinear transformations, moment matching, and exact edge-of-chaos analysis, we construct paired families of activation functions and initialization distributions—e.g., tanh with Uniform, SiLU with Gamma—that provably sustain Gaussian pre-activations over extended training in finite-width settings. Theoretical analysis establishes rigorous guarantees, and empirical validation confirms long-term Gaussian fidelity. All code is publicly released to support reproducibility and further extension.

This work proposes a novel approach to spatially localizing functional specialization in neural networks through structured noise, enabling a single network to efficiently store and distinguish multiple functions. By introducing a virtual noise field that generates spatially structured noise in a continuous auxiliary space, the method activates partially overlapping subnetworks and leverages cross-activation functions to achieve multi-level parameter sharing at the sample, statistical, and analytical levels. The key innovation lies in repurposing noise from a source of interference into an active regulatory mechanism that defines the topological structure of functional subnetworks. Experiments on one-dimensional function approximation demonstrate that memory capacity significantly increases when the spatial configuration of the noise field aligns with the similarity structure of target functions, while misalignment leads to degraded performance, revealing a critical relationship between noise structure and function representation.

This work elucidates the underlying mechanism of gated MLPs by offering the first explanation of their success through the lens of symmetry breaking. It demonstrates that a gated MLP can be interpreted as a rank-1 approximation of bilinear attention, where the query and key correspond to two distinct factors, and the nonlinear activation is applied exclusively to one factor. This asymmetric treatment breaks both the exchange symmetry between the two factors and the inverse scaling symmetry induced by non-homogeneous activation functions. The analysis establishes a theoretical connection between gated MLPs and attention mechanisms, clarifying the origin of their performance advantages and providing a principled foundation for designing novel, efficient architectures.

This study addresses the lack of general design principles linking neural network architecture to computational capacity. By systematically evaluating the computational performance of recurrent neural networks with diverse connectivity patterns on Boolean function tasks through large-scale sampling, the work reveals— for the first time—that local 2-cycles and 3-cycles are critical structural motifs for enhancing computational power. It further demonstrates that introducing a small number of sparse connections and biologically inspired interneuron-like units significantly boosts the performance of large-scale networks. The authors construct a comprehensive performance map linking small-network architectures to Boolean function realization, showing that networks containing short cycles achieve optimal performance. Moreover, network performance can be accurately predicted from structural statistics, offering a theoretical foundation and biologically inspired guidance for future neural architecture design.

Physics-Informed Neural Networks (PINNs) often struggle to train reliably on stiff and oscillatory dynamical systems due to poor optimization conditioning. While prior work has emphasized representational remedies such as spectral parameterizations, the optimization implications of initial-condition (IC) embeddings in adaptive spectral PINNs have not been well characterized. In this work, we show that the choice of IC gating function induces explicit time-dependent gradient scaling, which interacts with spectral representations during training. Using a nonlinear stiff spring-pendulum ODE as a controlled benchmark, we compare exponential and linear IC gates in combination with fixed and adaptive Fourier spectral trunks. We observe stiffness-dependent changes in relative dominance for adaptive PINNs: at moderate stiffness ($k=20$), exponential gating often yields lower error but exhibits heterogeneous behavior across random seeds, whereas at higher stiffness ($k=60$), linear gating becomes preferable, with additional reversals observed at larger $k$. These trends hold for both relative $L^2$ error and maximum pointwise error and are confirmed by paired Wilcoxon signed-rank tests with Holm correction. Overall, our results demonstrate that IC embeddings are not a neutral design choice in PINNs: the induced gradient scaling materially shapes optimization conditioning in stiff regimes, with distinct sensitivity patterns in baseline and adaptive spectral models.