A Production Scheduling Framework for Reinforcement Learning Under Real-World Constraints

2025-06-16

AI Summary

Real-world production scheduling confronts multiple dynamic constraints—including material handling logistics, buffer capacity limits, machine failures, setup times, and stochastic processing durations—posing challenges for both modeling generality and evaluation fairness in conventional methods and existing RL frameworks. This paper proposes the first modular, customizable RL-based scheduling framework that uniformly models heterogeneous uncertainties, supports multi-objective optimization, and enables flexible scenario configuration. Leveraging a high-fidelity discrete-event simulation environment as a digital twin, the framework integrates PPO/SAC algorithms, multi-objective reward shaping, enhanced state encoding, and robust training strategies. Evaluated using the open-source toolkit JobShopLab, the proposed RL policies achieve, under diverse disturbances, an average 18.7% reduction in weighted tardiness and a 12.3% improvement in overall equipment effectiveness (OEE) compared to rule-based schedulers.

Technology Category

• Planning, Routing, and Scheduling: Scheduling under Uncertainty
• Reasoning under Uncertainty: Stochastic Optimization
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

The classical Job Shop Scheduling Problem (JSSP) focuses on optimizing makespan under deterministic constraints. Real-world production environments introduce additional complexities that cause traditional scheduling approaches to be less effective. Reinforcement learning (RL) holds potential in addressing these challenges, as it allows agents to learn adaptive scheduling strategies. However, there is a lack of a comprehensive, general-purpose frameworks for effectively training and evaluating RL agents under real-world constraints. To address this gap, we propose a modular framework that extends classical JSSP formulations by incorporating key mbox{real-world} constraints inherent to the shopfloor, including transport logistics, buffer management, machine breakdowns, setup times, and stochastic processing conditions, while also supporting multi-objective optimization. The framework is a customizable solution that offers flexibility in defining problem instances and configuring simulation parameters, enabling adaptation to diverse production scenarios. A standardized interface ensures compatibility with various RL approaches, providing a robust environment for training RL agents and facilitating the standardized comparison of different scheduling methods under dynamic and uncertain conditions. We release JobShopLab as an open-source tool for both research and industrial applications, accessible at: https://github.com/proto-lab-ro/jobshoplab

Problem

• Extends JSSP with real-world constraints like transport and breakdowns
• Lacks general RL framework for dynamic production scheduling
• Needs standardized comparison of scheduling methods under uncertainty

Innovation

• Modular framework for RL scheduling
• Incorporates real-world production constraints
• Standardized interface for RL compatibility