🤖 AI Summary
This work addresses the limited expressiveness of fixed activation functions in neural networks and the substantial computational overhead introduced by existing learnable activations. We propose a lightweight, general-purpose parametric nonlinearity enhancement framework that dynamically modulates the nonlinear behavior of standard activation functions (e.g., ReLU) via an end-to-end differentiable, adaptive parameterization. This enables structural adaptability while improving model performance. The framework is plug-and-play, fully compatible with standard CNN architectures, and introduces only a negligible number of learnable parameters. Extensive experiments on MNIST and multiple CNN benchmark tasks demonstrate significant improvements in classification accuracy, with near-negligible increases in computational cost (<0.5% additional FLOPs). Empirical results validate its strong generalization capability and practical utility.
📝 Abstract
A general procedure for introducing parametric, learned, nonlinearity into activation functions is found to enhance the accuracy of representative neural networks without requiring significant additional computational resources. Examples are given based on the standard rectified linear unit (ReLU) as well as several other frequently employed activation functions. The associated accuracy improvement is quantified both in the context of the MNIST digit data set and a convolutional neural network (CNN) benchmark example.