Two-Timescale Hierarchical Reinforcement Learning for Resilient Operations

2026-07-25

AI Summary

This work proposes a two-timescale hierarchical reinforcement learning framework to enhance the resilience of global operations systems under sudden disruptions. By integrating long-term decisions (e.g., inventory replenishment) with short-term actions (e.g., dynamic pricing) within a hierarchical policy structure, the method achieves coordinated optimization across timescales. It provides the first convergence guarantee for coupled two-timescale learning, yielding an average policy gap of \(O(T^{-1/2})\), improvable to \(O(\log T / T)\) under specific conditions. Leveraging synchronized policy updates and an adaptive learning mechanism driven by profit feedback, the approach demonstrates superior performance in a used-car case study: it increases average profits by 9.2% and 11.8% over the strongest partially adaptive benchmark under joint supply-demand shocks and persistent disturbances, respectively, while substantially improving profit stability.

Abstract

Unexpected shocks recur in global operations, requiring decision rules that adapt as market and operating conditions change. Many operational systems also have hierarchical structures in which long-term and short-term decisions pursue a shared objective. We study how hierarchical reinforcement learning can strengthen resilience by adapting these interdependent rules jointly. We develop a two-timescale hierarchical reinforcement learning framework that adapts long-term and short-term policies at their respective time scales. Because the policies are interdependent, we synchronize their updates and prove, to our knowledge, the first convergence guarantees for coupled two-timescale learning. Over $T$ periods, our policies' average gap from an optimal policy pair is $O(T^{-1/2})$, improving to $O(\log T/T)$ when poor decisions produce clearer profit losses. In a used-car case study, inventory replenishment is the long-term decision and customer-arrival pricing the short-term decision. Relative to the strongest partially adaptive benchmark, the framework increases mean profit by $9.2\%$ under joint demand-supply shocks and by $11.8\%$ under a prolonged shock scenario, while maintaining a more stable profit trajectory over time. Short-term adaptation addresses routine seasonality and one-sided disruptions by responding immediately to changing conditions. Under joint demand-supply shocks, however, it is insufficient alone; long-term adaptation is also needed to create favorable conditions for short-term decisions. Joint adaptation thus yields higher and more stable profits through disruption and recovery. Because many organizations already use hierarchical planning, the framework strengthens operational resilience without altering existing decision structures.

Problem

• resilient operations
• hierarchical decision-making
• two-timescale adaptation
• demand-supply shocks
• operational resilience

Innovation

• two-timescale reinforcement learning
• hierarchical decision-making
• operational resilience
• convergence guarantees
• joint policy adaptation