On the Interplay Between Sparsity and Training in Deep Reinforcement Learning

2025-01-28

AI Summary

This work investigates the mechanistic impact of sparse architectural designs on the performance of image-input deep reinforcement learning (DRL). Addressing a critical gap—prior studies overlook how weight trainability modulates sparsity benefits—we systematically compare spatially biased, local convolutional, randomly sparse fully connected, and topologically constrained graph network architectures, all with matched parameter counts, under both fixed-weight and trainable-weight hidden-layer configurations. Evaluations are conducted within the DQN and IMPALA frameworks on the Atari and DeepMind Lab benchmarks. Our key finding is that the optimal sparse architecture is strongly contingent on weight trainability—a previously unrecognized coupling between sparsity gains and parameter optimization dynamics. Specifically, under trainable-weight conditions, certain sparse structures improve sample efficiency by up to 37%; conversely, the best-performing architecture shifts markedly when weights are frozen. This demonstrates that sparsity-induced improvements are not task-agnostic but fundamentally intertwined with gradient-based learning mechanisms.

Technology Category

• Machine Learning: Deep Neural Architectures and Foundation Models
• Search and Optimization: Learning to Search
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Graph neural networks and deep learning approaches for Web-related graphs
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We study the benefits of different sparse architectures for deep reinforcement learning. In particular, we focus on image-based domains where spatially-biased and fully-connected architectures are common. Using these and several other architectures of equal capacity, we show that sparse structure has a significant effect on learning performance. We also observe that choosing the best sparse architecture for a given domain depends on whether the hidden layer weights are fixed or learned.

Problem

• Studying sparse architectures' benefits in deep reinforcement learning
• Focusing on image-based domains with common architectures
• Investigating how sparse structure affects learning performance

Innovation

• Sparse architectures improve deep reinforcement learning
• Spatially-biased and fully-connected architectures compared
• Performance depends on fixed or learned weights