🤖 AI Summary
Traditional reinforcement learning (RL) suffers from poor generalization, weak robustness, and limited interpretability under distributional shifts, confounding variables, and dynamic environments due to its reliance on correlational modeling. To address these limitations, this paper proposes a Causal Reinforcement Learning (CRL) framework that systematically integrates causal inference (CI) techniques. We introduce the first unified taxonomy covering five core directions: causal representation learning, counterfactual policy optimization, offline causal RL, causal policy transfer, and causal world modeling. We formally define three foundational paradigms—causal representation, counterfactual policy optimization, and causal transfer—and uncover a novel pathway for causal disentanglement under distributional shift. Leveraging structural causal models (SCMs), do-calculus, invariant risk minimization (IRM), and counterfactual policy evaluation, we comprehensively survey over 100 state-of-the-art works. Empirical results demonstrate that CRL significantly improves cross-environment generalization and decision trustworthiness in robotics control, healthcare decision-making, and recommender systems.
📝 Abstract
Integrating causal inference (CI) with reinforcement learning (RL) has emerged as a powerful paradigm to address critical limitations in classical RL, including low explainability, lack of robustness and generalization failures. Traditional RL techniques, which typically rely on correlation-driven decision-making, struggle when faced with distribution shifts, confounding variables, and dynamic environments. Causal reinforcement learning (CRL), leveraging the foundational principles of causal inference, offers promising solutions to these challenges by explicitly modeling cause-and-effect relationships. In this survey, we systematically review recent advancements at the intersection of causal inference and RL. We categorize existing approaches into causal representation learning, counterfactual policy optimization, offline causal RL, causal transfer learning, and causal explainability. Through this structured analysis, we identify prevailing challenges, highlight empirical successes in practical applications, and discuss open problems. Finally, we provide future research directions, underscoring the potential of CRL for developing robust, generalizable, and interpretable artificial intelligence systems.