Forecasting from Counterfactual Simulator Rollouts: A Sim2Real Evaluation

2026-10-02

AI Summary

This study addresses the cold-start challenge in predictive modeling when deploying new decision policies, where historical data fails to reflect system responses. To overcome this, we propose a Sim2Real transfer framework based on counterfactual simulation. A simulator generates counterfactual trajectories to train predictive models for zero-shot transfer, complemented by a lightweight calibration mechanism leveraging early real-world observations to bridge the sim-to-real gap. The method is validated in a real-world inventory control scenario. Experimental results demonstrate that the simulation-trained model reduces Mean Absolute Percentage Error (MAPE) by 1.2–18.7 percentage points compared to historical data baselines, with calibration further decreasing errors by up to 2.5 percentage points. These findings establish an efficient and practical solution for cold-start prediction under novel decision strategies.

Abstract

Deploying a new decision policy creates a cold-start problem for prediction models whose targets depend on the policy's actions: historical observations reflect earlier policies, while real observations under the new policy are not yet available. Simulation offers a way to address this gap by rolling out the target policy across counterfactual scenarios and using the resulting trajectories to learn how the system responds to those controls. The simulation-to-reality (Sim2Real) transfer of this simulator-trained model can then be backtested by evaluating it against real observations from past deployments. Using two real-world inventory-control deployments, we evaluate this process from three angles: simulator fidelity, zero-shot transfer to real behavior, and adaptation as real target-policy observations accumulate. The simulator-trained forecaster achieves lower point-estimate mean absolute percentage error (MAPE) than the same architecture trained on historical real data, reducing MAPE by 1.2-3.1 percentage points in Study 1 and 12.5-18.7 points in Study 2. After deployment, lightweight calibration using early real observations further reduces error by up to 2.5 percentage points. These results provide empirical evidence that simulator-generated counterfactual data can support cold-start forecasting under a new policy, and the resulting model can be further refined as real deployment data become available.

Problem

• cold-start problem
• Sim2Real transfer
• counterfactual simulation
• policy deployment
• forecasting

Innovation

• Sim2Real
• Counterfactual Simulation
• Cold-start Forecasting
• Zero-shot Transfer
• Lightweight Calibration