Efficient Heteroscedastic Bayesian Optimization for Risk-Aware AutoRL

2026-07-29

AI Summary

This work addresses the limitation of existing reinforcement learning hyperparameter optimization methods, which often neglect performance instability and struggle to simultaneously achieve high average returns and low training variance. To overcome this, the authors propose ERAHBO, a novel approach that, for the first time in automated reinforcement learning (AutoRL), jointly models the influence of hyperparameters on both expected return and variance. ERAHBO employs heteroscedastic Bayesian optimization to enable risk-aware tuning and incorporates an adaptive resampling mechanism to enhance sample efficiency. Experimental results demonstrate that ERAHBO significantly outperforms both risk-neutral and risk-averse baselines across diverse algorithms and environments, consistently delivering high returns while effectively reducing variance, thereby improving tuning efficiency in risk-sensitive scenarios.

Abstract

Reinforcement learning (RL) has shown remarkable success across a wide range of complex tasks. However, RL outcomes can be highly stochastic, and both expected performance and variability often depend on hyperparameter (HP) configurations. We propose efficient and risk-averse heteroscedastic Bayesian Optimization (ERAHBO), a Bayesian optimization method that models both the mean and variance of learning outcomes as functions of the HP configurations. ERAHBO aims to identify HP configurations that achieve high average return while reducing variability across training runs, and it improves the sample efficiency of the HP optimization via adaptive re-sampling rather than a fixed budget per HP. Empirical evaluations across diverse RL algorithms and environments demonstrate that ERAHBO generally outperforms both risk-neutral and risk-averse baselines, delivering improved sample efficiency for risk-averse returns.

Problem

• Bayesian Optimization
• Heteroscedasticity
• Risk-Aware
• Hyperparameter Optimization
• Reinforcement Learning

Innovation

• Heteroscedastic Bayesian Optimization
• Risk-Aware AutoRL
• Adaptive Re-sampling
• Sample Efficiency
• Hyperparameter Optimization