Hadamard Representations: Augmenting Hyperbolic Tangents in RL

2024-06-13

AI Summary

This work reveals that continuously differentiable activation functions (e.g., tanh) suffer from pervasive neuron death and gradient vanishing in reinforcement learning, severely impairing representational capacity and training stability. To address this, we propose a Hadamard-product-based activation enhancement mechanism: a lightweight, plug-and-play module that applies learnable, element-wise modulation to dynamically amplify gradient flow, mitigate neuron deactivation, and increase the network’s effective rank. The method is architecture-agnostic, compatible with mainstream RL frameworks including DQN, PPO, and parallel Q-networks. Extensive evaluation on the Atari benchmark demonstrates that our approach significantly accelerates convergence, reduces the proportion of dead neurons by up to 42%, improves effective rank and policy stability, and delivers consistent performance gains across diverse algorithms. To our knowledge, this is the first solution enabling robust, reliable deployment of continuously differentiable activations in deep RL.

Technology Category

• Machine Learning: Reinforcement Learning
• Multiagent Systems: Adversarial Agents
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Activation functions are one of the key components of a deep neural network. The most commonly used activation functions can be classed into the category of continuously differentiable (e.g. tanh) and piece-wise linear functions (e.g. ReLU), both having their own strengths and drawbacks with respect to downstream performance and representation capacity through learning (e.g. measured by the number of dead neurons and the effective rank). In reinforcement learning, the performance of continuously differentiable activations often falls short as compared to piece-wise linear functions. We provide insights into the vanishing gradients associated with the former, and show that the dying neuron problem is not exclusive to ReLU's. To alleviate vanishing gradients and the resulting dying neuron problem occurring with continuously differentiable activations, we propose a Hadamard representation. Using deep Q-networks, proximal policy optimization and parallelized Q-networks in the Atari domain, we show faster learning, a reduction in dead neurons and increased effective rank.

Problem

• Dying neuron issue in RL with differentiable activations like tanh
• Performance gap between differentiable and piece-wise linear activations
• Need for better activation functions to enhance RL learning

Innovation

• Hadamard representation enhances hyperbolic tangent functions
• Solves dying neuron problem in differentiable activations
• Improves learning speed and reduces dead neurons