Hierarchical Constrained Reinforcement Learning with Dynamic Boundary for Spatio-Temporal Vehicle-to-Grid Scheduling

2026-08-04

AI Summary

This study addresses the challenges posed by large-scale electric vehicle (EV) integration into power grids, where spatiotemporal coupling constraints and computational complexity hinder existing approaches from simultaneously ensuring constraint satisfaction, computational efficiency, and scalability. To overcome these limitations, the authors propose a Hierarchical Constrained Reinforcement Learning framework (HPC-RL). The upper layer rigorously enforces grid-level spatial hard constraints using a generalized reduced gradient method, while the lower layer employs a dynamic boundary policy to generate real-time feasible charging/discharging regions for individual EVs, effectively decoupling spatial grid constraints from temporal charging demands. Evaluated on IEEE benchmark systems, the proposed method significantly outperforms model predictive control and state-of-the-art safe reinforcement learning baselines, achieving near-zero constraint violation rates, nearly 100% fulfillment of charging demands, and reducing online scheduling time for large-scale scenarios from hours to minutes.

Abstract

The rapid proliferation of Electric Vehicles (EVs) introduces significant spatio-temporal uncertainties into power grids, while Vehicle-to-Grid (V2G) technology offers critical flexibility through bidirectional power flow. However, integrating large-scale EVs into the Optimal Power Flow framework presents substantial challenges due to computational bottlenecks arising from solver complexity and coupled spatio-temporal constraints. Existing Reinforcement Learning (RL) methods often struggle to balance strict constraint satisfaction with scalability in highly dynamic EV fleet environments. To address these challenges, this paper proposes a Hierarchical Policy for Constrained Reinforcement Learning (HPC-RL) framework for spatially and temporally coupled V2G scheduling. The framework adopts a two-layer architecture: the upper level utilizes a RL algorithm based on the Generalized Reduced Gradient method to strictly enforce spatial grid-level hard constraints; the lower level implements a novel dynamic boundary strategy to compute real-time feasible charging power bounds for individual EVs, thereby ensuring the satisfaction of temporal charging demands. This integrated design not only enables the simultaneous handling of spatially and temporally coupled constraints during the RL optimization process but also significantly enhances generalization capabilities for large-scale fleets through hierarchical decoupling. Extensive experiments on IEEE 14, 30, and modified 141-bus systems demonstrate that HPC-RL outperforms Model Predictive Control and state-of-the-art safe RL baselines across all metrics. The proposed method achieves near-optimal scheduling strategies and drastically reduces online computation time in large-scale scenarios from hours to minutes, while maintaining a near-zero constraint violation rate and nearly 100\% charging demand satisfaction.

Problem

• Vehicle-to-Grid
• Spatio-Temporal Constraints
• Constrained Reinforcement Learning
• Optimal Power Flow
• Electric Vehicles

Innovation

• Hierarchical Reinforcement Learning
• Dynamic Boundary
• Spatio-Temporal Constraints
• Vehicle-to-Grid Scheduling
• Constrained Optimization