LayerAct: Advanced Activation Mechanism for Robust Inference of CNNs

2023-06-08

AI Summary

Existing CNN activation functions exhibit insufficient robustness to input noise, while layer normalization (LN) mitigates this issue at the cost of feature homogenization. To address this trade-off, we propose LayerAct—the first hierarchical activation function that safely integrates LN principles directly into the activation mechanism. LayerAct theoretically models the upper bound of output perturbation induced by input shifts, thereby overcoming the robustness limitations inherent in element-wise activations. It jointly leverages hierarchical statistical estimation and learnable scaling/bias parameters, preserving standard CNN training pipelines while simultaneously enhancing noise robustness and maintaining representational diversity. Evaluated on three image classification benchmarks—under both clean and noisy conditions—LayerAct consistently outperforms ReLU, GELU, and other baselines in robustness to label and input noise. Moreover, it achieves superior or comparable accuracy on clean data across most settings, demonstrating its effectiveness without compromising standard performance.

Abstract

In this work, we propose a novel activation mechanism called LayerAct for CNNs. This approach is motivated by our theoretical and experimental analyses, which demonstrate that Layer Normalization (LN) can mitigate a limitation of existing activation functions regarding noise robustness. However, LN is known to be disadvantageous in CNNs due to its tendency to make activation outputs homogeneous. The proposed method is designed to be more robust than existing activation functions by reducing the upper bound of influence caused by input shifts without inheriting LN's limitation. We provide analyses and experiments showing that LayerAct functions exhibit superior robustness compared to ElementAct functions. Experimental results on three clean and noisy benchmark datasets for image classification tasks indicate that LayerAct functions outperform other activation functions in handling noisy datasets while achieving superior performance on clean datasets in most cases.

Problem

• Enhances noise robustness in CNNs
• Improves activation function performance
• Outperforms in clean and noisy datasets

Innovation

• Introduces LayerAct for CNNs
• Enhances noise robustness
• Outperforms ElementAct functions