Performance Variation in Deep Reinforcement Learning

2026-06-04

AI Summary

This study addresses the substantial performance variability of deep reinforcement learning algorithms across repeated runs under identical settings—a phenomenon often inadequately captured by prevailing evaluation protocols that either underestimate this variance or misalign with practical objectives. To remedy this, the work proposes a percentile-based statistical and visualization framework, introducing the min-max Inter-Percentile Range (IPR) metric and a per-run percentile highlighting technique to more accurately characterize both inter-algorithm and intra-algorithm performance differences. Systematic experiments across PPO, SAC, TD-MPC variants, DQN, and Rainbow reveal that LayerNorm effectively reduces PPO’s variability, TD-MPC exhibits the lowest variability and highest data efficiency among tested algorithms, and DQN and Rainbow demonstrate comparable levels of variability on Atari benchmarks.

Abstract

Deep reinforcement learning (RL) algorithms often suffer from low run-to-run robustness, manifesting as significant performance variation across independent runs of identically configured agents. Although this issue poses a spectrum of challenges across research and practice, relatively few studies develop methods to evaluate it; RL research instead often reports uncertainty in the estimated mean performance. In this paper, we outline the limitations of conventional uncertainty and variation estimates, particularly their misalignment with purpose and the risk of underreporting. We then propose an alternative percentile-based statistic and visualization method, min-max IPR and run-wise percentile highlighting, respectively. These percentile-based tools are easy to interpret and rely on standard properties of sample percentiles, providing rich information about run-to-run performance variation. We demonstrate this through three case studies. First, we show that LayerNorm and penultimate-layer normalizations narrow performance variation in PPO, whereas the variation is mostly unchanged in SAC. Second, we compare PPO, SAC, TD-MPC, and TD-MPC2, and show TD-MPC exhibits the least variation while being the most data efficient among the four. Finally, in a comparison of DQN and Rainbow on five Atari environments, we show that both algorithms exhibit similar levels of performance variation.

Problem

• performance variation
• deep reinforcement learning
• run-to-run robustness
• uncertainty estimation
• reproducibility

Innovation

• performance variation
• percentile-based statistics
• min-max IPR
• run-wise visualization
• deep reinforcement learning robustness