🤖 AI Summary
In continuous control tasks, modeling only the mean of the state-action value function leads to insufficient policy robustness. This work first observes that the state-action value distribution is highly approximately Gaussian. Leveraging this insight, we propose Normal Quantile Distributional Reinforcement Learning (NQRL): a lightweight variance network estimates the distribution’s standard deviation; Gaussian target quantiles are derived in closed form; and a novel policy update rule is designed to enforce distributional structural consistency. NQRL avoids ensemble-based uncertainty estimation, substantially reducing both parameter count and training overhead. Evaluated on 16 standard continuous control benchmarks, NQRL achieves statistically significant performance improvements on 10 tasks, while converging faster and requiring fewer parameters than state-of-the-art ensemble-based distributional RL methods.
📝 Abstract
Learning a predictive model of the mean return, or value function, plays a critical role in many reinforcement learning algorithms. Distributional reinforcement learning (DRL) has been shown to improve performance by modeling the value distribution, not just the mean. We study the value distribution in several continuous control tasks and find that the learned value distribution is empirical quite close to normal. We design a method that exploits this property, employ variances predicted from a variance network, along with returns, to analytically compute target quantile bars representing a normal for our distributional value function. In addition, we propose a policy update strategy based on the correctness as measured by structural characteristics of the value distribution not present in the standard value function. The approach we outline is compatible with many DRL structures. We use two representative on-policy algorithms, PPO and TRPO, as testbeds. Our method yields statistically significant improvements in 10 out of 16 continuous task settings, while utilizing a reduced number of weights and achieving faster training time compared to an ensemble-based method for quantifying value distribution uncertainty.