CaRe-BN: Precise Moving Statistics for Stabilizing Spiking Neural Networks in Reinforcement Learning

2025-09-28

AI Summary

In online reinforcement learning (RL), spiking neural networks (SNNs) critically rely on batch normalization (BN) to stabilize training due to the discrete, non-differentiable nature of spike events. However, RL’s inherently dynamic and non-stationary data distribution induces biased BN statistics estimation, leading to gradient instability, slow convergence, and suboptimal policies. To address this, we propose CaRe-BN—the first BN optimization method tailored for SNNs in RL. CaRe-BN introduces confidence-guided adaptive sliding-window statistics update and hidden-layer activation distribution recalibration, enabling accurate and robust BN statistics estimation without incurring additional inference overhead. Evaluated on continuous control benchmarks, CaRe-BN achieves an average performance gain of 22.6% over baseline SNN-RL methods, outperforms ANN-based baselines by up to 5.9% on select tasks, significantly accelerates convergence, and improves policy quality—thereby advancing the efficient deployment of energy-efficient SNN agents on resource-constrained devices.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Online Learning & Bandits
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web

Abstract

Spiking Neural Networks (SNNs) offer low-latency and energy-efficient decision-making on neuromorphic hardware by mimicking the event-driven dynamics of biological neurons. However, due to the discrete and non-differentiable nature of spikes, directly trained SNNs rely heavily on Batch Normalization (BN) to stabilize gradient updates. In online Reinforcement Learning (RL), imprecise BN statistics hinder exploitation, resulting in slower convergence and suboptimal policies. This challenge limits the adoption of SNNs for energy-efficient control on resource-constrained devices. To overcome this, we propose Confidence-adaptive and Re-calibration Batch Normalization (CaRe-BN), which introduces (emph{i}) a confidence-guided adaptive update strategy for BN statistics and (emph{ii}) a re-calibration mechanism to align distributions. By providing more accurate normalization, CaRe-BN stabilizes SNN optimization without disrupting the RL training process. Importantly, CaRe-BN does not alter inference, thus preserving the energy efficiency of SNNs in deployment. Extensive experiments on continuous control benchmarks demonstrate that CaRe-BN improves SNN performance by up to $22.6%$ across different spiking neuron models and RL algorithms. Remarkably, SNNs equipped with CaRe-BN even surpass their ANN counterparts by $5.9%$. These results highlight a new direction for BN techniques tailored to RL, paving the way for neuromorphic agents that are both efficient and high-performing.

Problem

• Imprecise BN statistics hinder SNN exploitation in RL
• CaRe-BN stabilizes SNN optimization via adaptive normalization
• Method enables efficient neuromorphic control without altering inference

Innovation

• Confidence-guided adaptive update for BN statistics
• Re-calibration mechanism to align distributions
• Preserves SNN energy efficiency during inference