Treatment Effect Modification by Latent Classes

2026-10-07

AI Summary

This study addresses the challenge of identifying heterogeneous causal effects on digital platforms, where users' latent states—such as emotional conditions—are unobservable. To overcome this, we propose an estimation framework integrating Hidden Markov Models (HMMs) with Neyman orthogonalization. Specifically, the HMM infers unobserved states serving as effect modifiers, while an orthogonalized estimator is constructed to mitigate HMM fitting errors, enabling standard confidence interval construction and robust identification of state-dependent effects. We validate this approach using large-scale A/B testing data from Netflix. Our findings reveal that highly engaged users derive significant benefits, whereas low-activity or repetitive-viewing users experience limited gains. Ultimately, this work establishes a novel paradigm for platforms to precisely evaluate experimental interventions under latent state heterogeneity.

Abstract

Many treatments have effects that plausibly vary by latent classes. For example, the effect of personalized video recommendations may be stronger for users who seek a new discovery than for users seeking to rewatch something familiar. In the context of user journeys on digital platforms, we can conceptualize user moods as dynamic latent states that lead to distinct action distributions. Because mood is unobserved, we infer it from a user's pre-treatment actions using a hidden Markov model (HMM), and then leverage the latent state as an interpretable treatment effect modifier when evaluating experiment results. Under a treatment-effect sufficiency assumption, each observable conditional effect is a known mixture of the latent effects, so state-specific effects are identified when state posteriors are sufficiently heterogeneous. We propose an estimator that is Neyman-orthogonal to errors in the fitted HMM, enabling standard influence-function confidence intervals. In a real-world large-scale A/B test of a new recommendation model at Netflix, our latent-state analysis reveals that heavy-engagement users show significantly positive effects across all treatment variants, with limited evidence of benefit among less-active and rewatch-oriented users.

Problem

• treatment effect modification
• latent classes
• hidden Markov model
• heterogeneous treatment effects
• unobserved states

Innovation

• Latent Classes
• Hidden Markov Model
• Treatment Effect Modification
• Neyman-orthogonality
• Causal Inference