Technology Category
Cognitive Modeling & Cognitive Systems: Neural Spike CodingMachine Learning: Deep Neural Architectures and Foundation ModelsSearch and Optimization: Learning to Search
Improving Performance of Spike-based Deep Q-Learning using Ternary Neurons
🤖 AI Summary
To address the severe performance degradation of ternary spiking neurons in deep Q-learning, this work identifies gradient estimation bias as the root cause and proposes a low-bias ternary spiking neuron model. The model enhances temporal decision-making representation and training stability by refining both the spike-generation mechanism and the gradient approximation strategy. Integrated with spike-based backpropagation and the Deep Spiking Q-Network (DSQN) architecture, it is systematically evaluated on seven Atari Gym games. Compared to binary baselines, the proposed method achieves a 23.6% average score improvement and fully mitigates the catastrophic performance drop previously observed in ternary spiking models. Experimental results demonstrate its feasibility and superiority for deploying reinforcement learning policies in resource-constrained edge intelligence scenarios.
Technology Category
Application Category
📝 Abstract
We propose a new ternary spiking neuron model to improve the representation capacity of binary spiking neurons in deep Q-learning. Although a ternary neuron model has recently been introduced to overcome the limited representation capacity offered by the binary spiking neurons, we show that its performance is worse than that of binary models in deep Q-learning tasks. We hypothesize gradient estimation bias during the training process as the underlying potential cause through mathematical and empirical analysis. We propose a novel ternary spiking neuron model to mitigate this issue by reducing the estimation bias. We use the proposed ternary spiking neuron as the fundamental computing unit in a deep spiking Q-learning network (DSQN) and evaluate the network's performance in seven Atari games from the Gym environment. Results show that the proposed ternary spiking neuron mitigates the drastic performance degradation of ternary neurons in Q-learning tasks and improves the network performance compared to the existing binary neurons, making DSQN a more practical solution for on-board autonomous decision-making tasks.
Problem
Research questions and friction points this paper is trying to address.
Enhancing representation capacity of binary spiking neurons in deep Q-learning
Addressing gradient estimation bias in ternary spiking neuron training
Improving performance of deep spiking Q-learning networks in Atari games
Innovation
Methods, ideas, or system contributions that make the work stand out.
Ternary spiking neuron model reduces estimation bias
Deep spiking Q-learning network (DSQN) enhances performance
Improved representation capacity over binary spiking neurons
🔎 Similar Papers
No similar papers found.
