Cumulative Treatment Effect Testing under Continuous Time Reinforcement Learning

2026-02-02

AI Summary

This study addresses the challenge of evaluating the cumulative treatment effect—encompassing both immediate and delayed components—under irregularly spaced observational times. The authors propose a novel approach based on continuous-time reinforcement learning, defining the average treatment effect (ATE) as the difference in infinite-horizon value functions and establishing its asymptotic normality to enable valid statistical inference. This work represents the first application of continuous-time reinforcement learning to causal effect estimation, offering improved accuracy in capturing dynamic treatment effects from short-term, irregularly sampled data compared to conventional discrete-time models. Empirical evaluations on multi-resolution synthetic scenarios and real-world data from the OhioT1DM dataset demonstrate the method’s effectiveness in assessing the cumulative impact of mealtime insulin on blood glucose levels.

Technology Category

• Machine Learning: Causal Learning
• Reasoning under Uncertainty: Causality
• Cognitive Modeling & Cognitive Systems: Conceptual Inference and Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Human-perceived consequences of algorithmic deployment on the web
• User Modeling, Personalization and Recommendation: Studies of user behavior, including longitudinal effects of personalized systems

Abstract

Understanding the impact of treatment effect over time is a fundamental aspect of many scientific and medical studies. In this paper, we introduce a novel approach under a continuous-time reinforcement learning framework for testing a treatment effect. Specifically, our method provides an effective test on carryover effects of treatment over time utilizing the average treatment effect (ATE). The average treatment effect is defined as difference of value functions over an infinite horizon, which accounts for cumulative treatment effects, both immediate and carryover. The proposed method outperforms existing testing procedures such as discrete time reinforcement learning strategies in multi-resolution observation settings where observation times can be irregular. Another advantage of the proposed method is that it can capture treatment effects of a shorter duration and provide greater accuracy compared to discrete-time approximations, through the use of continuous-time estimation for the value function. We establish the asymptotic normality of the proposed test statistics and apply it to OhioT1DM diabetes data to evaluate the cumulative treatment effects of bolus insulin on patients'glucose levels.

Problem

• treatment effect
• cumulative effect
• continuous-time reinforcement learning
• carryover effect
• average treatment effect

Innovation

• continuous-time reinforcement learning
• average treatment effect
• cumulative treatment effect
• carryover effect
• value function estimation