🤖 AI Summary
This study addresses the critical need for timely detection of performance degradation following marketing interventions, a task for which the sensitivity of existing Bayesian causal impact models in early monitoring remains unclear. The authors develop a simulation-based evaluation framework that systematically assesses the models’ alerting capability under varying effect sizes, confidence levels, and monitoring windows by perturbing daily traffic data along an abandonment-recovery marketing journey. Their work extends causal impact analysis from retrospective estimation to real-time monitoring, introducing a quantifiable method to evaluate detection sensitivity and revealing distinct behavioral patterns between proportional and sustained alerting criteria in dynamic environments. Results demonstrate that detection performance is highly dependent on the interaction among these three factors, with sustained criteria exhibiting greater stability and operational practicality over longer monitoring windows.
📝 Abstract
Marketing systems frequently undergo operational changes that may affect performance, making timely detection of adverse effects essential for decision making. While Bayesian Causal Impact models are widely used to estimate the causal effects of interventions, their ability to support early operational monitoring remains less explored. This paper proposes a simulation based framework to evaluate the sensitivity and detection capabilities of Bayesian causal impact analysis under controlled performance degradations. Using daily traffic data from an abandoned cart marketing journey, we repeatedly perturb the observed outcomes and assess alarm activation probabilities across different effect magnitudes, confidence levels, and detection horizons. Two alarm criteria are analyzed: one based on the proportion of observations falling below the predictive lower bound and another based on consecutive days of negative cumulative impact. Results show that detection performance depends strongly on the interaction between effect size, confidence level, and evaluation horizon. In particular, proportion based criteria become less effective as the monitoring horizon increases, whereas persistence based criteria provide more stable and operationally meaningful detection behavior. The proposed framework extends causal impact analysis beyond retrospective effect estimation, offering a practical methodology for quantifying detection sensitivity and supporting monitoring decisions in dynamic marketing environments.