Policy-Based Reinforcement Learning with Action Masking for Dynamic Job Shop Scheduling under Uncertainty: Handling Random Arrivals and Machine Failures

2026-01-14

AI Summary

This work proposes a scheduling approach for dynamic job shop scheduling problems under uncertainty caused by stochastic job arrivals and unexpected machine failures. The randomness of job arrivals and machine breakdowns is modeled using Gamma and Weibull distributions, respectively. To ensure policy optimization remains within the feasible action space, two action-masking mechanisms—non-gradient and gradient-based—are integrated with a Maskable Proximal Policy Optimization algorithm. Evaluated on standard dynamic JSSP benchmarks, the proposed method significantly outperforms conventional heuristics and dispatching rules, demonstrating superior performance in minimizing makespan, along with strong robustness and good scalability.

Technology Category

• Planning, Routing, and Scheduling: Scheduling under Uncertainty
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Mixed Discrete/Continuous Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Trust and reliance of crowd workers and data experts on GenAI
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

We present a novel framework for solving Dynamic Job Shop Scheduling Problems under uncertainty, addressing the challenges introduced by stochastic job arrivals and unexpected machine breakdowns. Our approach follows a model-based paradigm, using Coloured Timed Petri Nets to represent the scheduling environment, and Maskable Proximal Policy Optimization to enable dynamic decision-making while restricting the agent to feasible actions at each decision point. To simulate realistic industrial conditions, dynamic job arrivals are modeled using a Gamma distribution, which captures complex temporal patterns such as bursts, clustering, and fluctuating workloads. Machine failures are modeled using a Weibull distribution to represent age-dependent degradation and wear-out dynamics. These stochastic models enable the framework to reflect real-world manufacturing scenarios better. In addition, we study two action-masking strategies: a non-gradient approach that overrides the probabilities of invalid actions, and a gradient-based approach that assigns negative gradients to invalid actions within the policy network. We conduct extensive experiments on dynamic JSSP benchmarks, demonstrating that our method consistently outperforms traditional heuristic and rule-based approaches in terms of makespan minimization. The results highlight the strength of combining interpretable Petri-net-based models with adaptive reinforcement learning policies, yielding a resilient, scalable, and explainable framework for real-time scheduling in dynamic and uncertain manufacturing environments.

Problem

• Dynamic Job Shop Scheduling
• Uncertainty
• Random Arrivals
• Machine Failures
• Reinforcement Learning

Innovation

• Action Masking
• Coloured Timed Petri Nets
• Dynamic Job Shop Scheduling
• Maskable Proximal Policy Optimization
• Uncertainty Modeling