Technology Category
Planning, Routing, and Scheduling: Learning for Planning and SchedulingSearch and Optimization: Learning to SearchConstraint Satisfaction and Optimization: Constraint Learning and Acquisition
Electric Bus Charging Schedules Relying on Real Data-Driven Targets Based on Hierarchical Deep Reinforcement Learning
🤖 AI Summary
This paper addresses the joint optimization of cost minimization and constraint satisfaction in electric bus charging scheduling. We propose a multi-timescale hierarchical deep reinforcement learning framework: the upper layer models the problem as a Semi-Markov Decision Process (Semi-MDP), where charging objectives serve as decision units; the lower layer performs real-time power allocation using a dual Deep Q-Network (DQN) integrated with Hindsight Experience Replay (HER). We theoretically prove that this hierarchical policy is equivalent to the optimal solution of the original MDP, effectively mitigating learning challenges arising from long-horizon sparse rewards and non-stationary environments. Experiments on real-world operational data demonstrate a significant reduction in charging costs, 100% satisfaction of hard constraints—including range and timetable requirements—and a 3.2× improvement in training efficiency.
Technology Category
Application Category
📝 Abstract
The charging scheduling problem of Electric Buses (EBs) is investigated based on Deep Reinforcement Learning (DRL). A Markov Decision Process (MDP) is conceived, where the time horizon includes multiple charging and operating periods in a day, while each period is further divided into multiple time steps. To overcome the challenge of long-range multi-phase planning with sparse reward, we conceive Hierarchical DRL (HDRL) for decoupling the original MDP into a high-level Semi-MDP (SMDP) and multiple low-level MDPs. The Hierarchical Double Deep Q-Network (HDDQN)-Hindsight Experience Replay (HER) algorithm is proposed for simultaneously solving the decision problems arising at different temporal resolutions. As a result, the high-level agent learns an effective policy for prescribing the charging targets for every charging period, while the low-level agent learns an optimal policy for setting the charging power of every time step within a single charging period, with the aim of minimizing the charging costs while meeting the charging target. It is proved that the flat policy constructed by superimposing the optimal high-level policy and the optimal low-level policy performs as well as the optimal policy of the original MDP. Since jointly learning both levels of policies is challenging due to the non-stationarity of the high-level agent and the sampling inefficiency of the low-level agent, we divide the joint learning process into two phases and exploit our new HER algorithm to manipulate the experience replay buffers for both levels of agents. Numerical experiments are performed with the aid of real-world data to evaluate the performance of the proposed algorithm.
Problem
Research questions and friction points this paper is trying to address.
Optimizing electric bus charging schedules using hierarchical reinforcement learning
Minimizing charging costs while meeting daily operational targets
Overcoming long-range multi-phase planning with sparse rewards
Innovation
Methods, ideas, or system contributions that make the work stand out.
Hierarchical DRL decouples MDP into SMDP and MDPs
HDDQN-HER algorithm solves multi-temporal decision problems
Two-phase learning with HER enhances policy efficiency
🔎 Similar Papers
No similar papers found.
