Adaptive Finite-Budget Training for CVaR Risk-Aware Q-Learning

2026-08-04

AI Summary

This work addresses the instability in value estimation, high Bellman residuals, and low sample efficiency that plague CVaR risk-sensitive Q-learning under fixed hyperparameters. To overcome these limitations while preserving the original CVaR optimization objective, the authors propose an adaptive training controller that integrates six synergistic mechanisms: adaptive inner-loop step-size adjustment, synchronized decay of inner and outer loops, early correction of VaR variables, coverage-prioritized re-greedy sampling, progressive estimate aggregation, and data-driven scale calibration. Evaluated on a Bitcoin trading task, the proposed method reduces average Bellman residuals by 85%, achieves a policy Sharpe ratio of 0.9281, limits maximum drawdown to merely 6.46%, and demonstrates significantly lower volatility compared to the buy-and-hold strategy.

Abstract

Risk-aware Q-learning (RaQL) provides a model-free, two-timescale estimator for dynamic risk objectives, but its finite-budget behavior remains fragile: fixed inner-loop hyperparameters can produce unstable value estimates, persistent Bellman residuals, and inefficient sample reuse. This paper proposes an adaptive training controller for Conditional Value-at-Risk (CVaR) RaQL and evaluates it on a daily Bitcoin trading task. The controller preserves the original CVaR estimator and Bellman fixed point; instead, it redesigns the training procedure through six coordinated mechanisms: per-cell inner-step sizing, outer-rate-matched decay synchronization, a short early correction for the VaR-like inner variable, a coverage-first-then-greedy sample allocation rule, progressive suffix aggregation of mature inner estimates, and data-driven calibration of key scales from online-observable quantities. Across 20 random seeds and 856,000 inner-transition samples, the controller reduces the mean empirical CVaR Bellman residual by approximately 85% relative to the fixed-parameter baseline (MeanBEQ: 1.2202 to 0.1854; MeanBEV: 1.1624 to 0.0535) and maintains stability across CVaR levels, discount factors, and training budgets. On the chronological out-of-sample test set, the learned policy attains a Sharpe ratio of 0.9281 with a maximum drawdown of 6.46% after transaction costs. Although buy-and-hold yields a higher cumulative return (35.43% vs. 23.61%), the adaptive policy achieves far lower volatility (9.57% vs. 47.93%), drawdown, and CVaR loss. These results demonstrate that adaptive finite-budget training design, applied solely to the training procedure without altering the risk objective, can materially improve the reliability and risk-adjusted performance of risk-aware Q-learning in financial applications.

Problem

• Risk-aware Q-learning
• Finite-budget training
• CVaR
• Bellman residual
• Sample efficiency

Innovation

• adaptive training
• CVaR
• risk-aware Q-learning
• finite-budget optimization
• Bellman residual reduction