Hard Constraints, Smooth Gradients: Learning Feasible Inventory Policies via Differentiable Projection

2026-08-03

AI Summary

This work addresses stochastic sequential decision-making under hard constraints and combinatorial action spaces, where existing methods struggle to simultaneously ensure scalability and strict feasibility. The authors propose embedding differentiable convex optimization within the policy network: a neural network outputs continuous action targets, which are projected via quadratic programming onto a relaxed feasible set, and dual information is leveraged to map these projections to integer solutions that guarantee constraint satisfaction, enabling end-to-end training. This approach is the first to achieve full coverage of the action space under interactive hard constraints, with provable bounds on integer projection error, thereby combining the expressive power of mixed-integer linear programming (MILP) with the scalability of deep reinforcement learning. Experiments show an average optimality gap below 1% on small instances; on large-scale networks, it outperforms state-of-the-art base-stock policies by up to 9.75% and rolling-horizon stochastic programming by at least 7.7%; in an ASML industrial case study, it reduces costs by up to 3.22%.

Abstract

Many operational problems are constrained sequential decision processes with large, combinatorial action spaces and interdependent feasibility constraints. Mixed-integer linear programs (MILPs) handle such constraints flexibly but scale poorly in stochastic environments. Deep reinforcement learning (DRL) promises scalable decision rules, but existing methods either penalize constraints rather than enforce them, or rely on feasibility mechanisms that break down once constraints interact. We bridge this gap by embedding a differentiable convex optimization module inside the policy: a neural network proposes continuous action targets, a quadratic program projects them onto the relaxed feasible set, and a dual-informed integer mapping restores integrality while preserving feasibility. Given a differentiable simulator, the policy trains end to end from sampled trajectories using pathwise gradients, while handling hard constraints with similar flexibility to MILPs. We show that our feasibility enforcement has bounded error relative to an exact integer projection and ensures the entire feasible action space is reachable. We apply the method to multi-echelon production-inventory planning under shared resource and material constraints. Our policy attains an average optimality gap below 1% on small instances. It further outperforms state-of-the-art echelon base-stock policies by up to 9.75% and a rolling-horizon multi-stage stochastic program by at least 7.7% in larger networks. On an industry-scale case study from ASML, it reduces average cost by up to 3.22% relative to the best-known benchmark policy. The savings are largest where planning is hardest: in tightly capacitated systems with high demand variability. More broadly, our work shows that DRL can deliver economically significant savings in sequential decision problems with interdependent hard constraints, which are widespread in practice.

Problem

• hard constraints
• inventory policies
• sequential decision making
• combinatorial action spaces
• feasibility enforcement

Innovation

• differentiable projection
• hard constraints
• deep reinforcement learning
• feasible policy learning
• multi-echelon inventory