Physics-Constrained Soft Actor-Critic for Simulator-in-the-Loop Petroleum Reservoir History Matching

2026-08-03

AI Summary

This work addresses scientific calibration problems—such as reservoir history matching—that rely on computationally expensive full-physics simulators, lack gradient information, and suffer from limited data availability. The authors propose a reinforcement learning–driven in-simulator calibration paradigm, wherein the CMG IMEX simulator is wrapped as a Gymnasium environment and the soft actor-critic (SAC) algorithm learns a physics-constrained, stochastic multi-parameter proposal policy. Operating under a strict budget of only 200 simulator calls, the method efficiently explores the solution space without replacing the simulator, thereby guaranteeing that all candidate solutions undergo full-physics validation and balancing exploration efficiency with physical plausibility. Evaluated on the PUNQ-S3 benchmark, the approach achieves a macro-level NMSE of 0.0285, R² of 0.9324, and a matching score of 97.23%, while also uncovering well-level water production mismatches obscured by aggregate metrics.

Abstract

Many scientific calibration problems expose only an expensive executable simulator, making gradients unavailable and large-scale training-data generation impractical. We study petroleum reservoir history matching as an instance of this broader AI problem and formulate it as physics-constrained, simulator-in-the-loop policy search. Our method wraps the CMG IMEX full-physics simulator as a Gymnasium environment and uses Soft Actor-Critic (SAC) to learn a stochastic proposal distribution over continuous porosity, directional-permeability, and well-skin parameters. Each interaction generates and executes a reservoir case, aligns simulated and observed production responses, and returns a reward that combines multi-response mismatch with penalties for physically invalid properties. Off-policy replay reuses costly simulator feedback, while maximum-entropy learning preserves exploration. Unlike forward-surrogate approaches, the policy learns where to evaluate rather than learning to replace the simulator; every retained candidate is validated by IMEX. Under a 200-call budget on PUNQ-S3, the best valid candidate achieves category-macro NMSE $0.0285$, $R^2=0.9324$, and a bounded match score of $97.23\%$. Well-level analysis further exposes localized water-rate failures hidden by pooled metrics. These results establish a full-physics proof of concept for reinforcement-learning-based calibration of expensive scientific simulators.

Problem

• history matching
• expensive simulator
• gradient-free calibration
• physics-constrained optimization
• reservoir simulation

Innovation

• physics-constrained reinforcement learning
• simulator-in-the-loop
• Soft Actor-Critic
• history matching
• reservoir simulation