causal reinforcement learning

Reinforcement learning and causal inference have long remained disconnected, lacking a systematic integration of their shared counterfactual structure, which limits agents’ generalization and reasoning capabilities in complex environments. This work proposes the first unified causal reinforcement learning framework by modeling the environment as a structural causal model, thereby uncovering the implicit causal mechanisms underlying reinforcement learning. The framework seamlessly integrates online learning, off-policy evaluation, and causal calculus. Building on this foundation, it further extends to novel tasks such as intervention selection, imitation learning, and counterfactual policy learning. This study establishes a theoretical basis for merging causality with reinforcement learning, offering a new pathway toward intelligent decision-making that is generalizable, interpretable, and cross-modal.

Reinforcement learning (RL) struggles to discover and leverage causal relationships, resulting in limited interpretability and suboptimal decision-making efficiency. Method: This paper proposes the first online causal RL framework that tightly couples causal graph modeling, active intervention learning, and policy optimization in a closed loop. Contributions/Results: (1) We introduce the first alternating optimization mechanism for dynamic causal structure learning and policy refinement; (2) we develop the first fault-alert simulation benchmark enabling direct interventions in state space; (3) we provide theoretical guarantees showing that causal guidance induces a positive feedback loop for performance improvement. Experiments demonstrate significant gains over state-of-the-art methods on root-cause localization tasks, with enhanced robustness and interpretability. The code is publicly available.

This work addresses the challenge of explaining reinforcement learning policies through causal attribution of success and failure. To tackle the difficulty of attributing causality in high-dimensional, nonlinear agent–environment interactions, we propose the first intervention-consistent nonlinear causal model reduction framework. It stimulates causal responses among states, actions, and rewards via randomized action perturbations, then jointly applies nonlinear dimensionality reduction and causal structure learning to automatically extract high-level behavioral patterns with explicit causal semantics from raw trajectories. We theoretically prove that, under generalized additive models, this framework admits a unique exact solution. Evaluated on benchmark tasks—including cart-pole control and robotic table tennis—the method successfully identifies policy biases, critical decision bottlenecks, and failure mechanisms, outperforming existing black-box explanation approaches by a significant margin.

Traditional model-based reinforcement learning (MBRL) suffers from poor robustness to distributional shifts and limited generalization due to its failure to capture the underlying causal mechanisms of environment dynamics, leading it to learn spurious correlations. To address this, we propose Causal Model-based Policy Optimization (C-MBPO), the first MBRL framework integrating Structural Causal Models (SCMs) to formulate an intervenable and interpretable Causal Markov Decision Process (C-MDP). C-MBPO jointly performs online trajectory learning, local SCM identification, causal Bayesian network inference, counterfactual state–reward simulation, and intervention-aware policy gradient optimization. Experiments demonstrate that C-MBPO significantly improves policy robustness and out-of-distribution generalization under both near- and far-domain distribution shifts. Crucially, it accurately detects and suppresses spurious correlations, yielding stable, interpretable, and causally grounded decision-making.

Reinforcement learning (RL) faces the “curse of dimensionality” in high-dimensional tasks, while conventional curriculum learning relies on the unrealistic assumption that source and target tasks share an invariant optimal policy. Method: This paper introduces “causal alignment” from a causal inference perspective to ensure transferability of optimal decision rules between source and target tasks. Based on causal graph models, we derive the first sufficient condition for causal alignment. Our framework integrates causal modeling, counterfactual reasoning, and an adaptive curriculum generation algorithm, supporting both discrete and continuous action spaces as well as pixel-level observations. Results: Experiments demonstrate that our approach significantly improves policy transfer efficiency and final performance under unobserved confounding—outperforming standard curriculum learning in robustness and convergence speed.

This work addresses the challenge that world models in multi-agent systems often conflate statistical correlations with causal mechanisms, leading to performance degradation under distributional shifts. The authors propose an implicit causal world model that does not require a predefined causal graph and instead leverages only offline multi-agent demonstrations. By introducing policy variance in partially observable environments, the approach ensures that environmental dynamics satisfy the sequential backdoor criterion, thereby enabling identifiability of the underlying causal structure. Integrating implicit causal modeling, the sequential backdoor criterion, and offline reinforcement learning, the method learns interpretable causal representations across tasks such as Two-Door, Navigation, and Giveaway. Notably, model accuracy improves significantly with increasing intervention strength, demonstrating the efficacy of the proposed framework in capturing true causal relationships.

While existing large language model–based social simulations can generate realistic interactions, they lack causal semantics, limiting their ability to support reliable causal inference for governance interventions. This work introduces, for the first time, a systematic integration of necessity and sufficiency–based causal concepts into social simulation, establishing a counterfactual framework tailored for policy evaluation. The framework explicitly articulates the relationship between simulator fidelity and policy relevance, offering theoretical guidance for simulator design and defining the fidelity criteria necessary to support valid policy inferences. By doing so, it advances social simulation beyond mere plausibility toward genuine decision-support capability.

This work addresses the challenge of efficiently designing sequential interventional experiments in causal discovery to maximize information gain while incorporating prior knowledge. The authors propose a preference-based sequential experimental design framework that formulates intervention strategies as a sequential decision-making process. By learning relative preferences between pairs of interventions—rather than absolute rewards—the approach enables adaptive planning through Direct Preference Optimization (DPO). This method leverages preference data to train policies without relying on non-stationary reward signals and autonomously learns theoretically grounded intervention patterns, such as focusing on parent variables. Experiments on synthetic, physical simulation, and economic datasets demonstrate that, under identical intervention budgets, the proposed method achieves a 70–71% improvement over baseline approaches (p < 0.001, Cohen’s d ≈ 2).

This work proposes a novel approach that integrates causal inference into diffusion-based reinforcement learning, addressing the limitation of existing diffusion policies that rely solely on statistical correlations and thus fail to identify action components with genuine causal effects on returns. The method first jointly learns a base diffusion policy and a causal dynamics model from offline data, then continuously refines the causal structure through online interaction, leveraging a causally guided mechanism to optimize action generation. By unifying causal discovery, diffusion probabilistic modeling, and offline reinforcement learning, the approach achieves significantly superior performance—both in terms of efficacy and stability—over current state-of-the-art methods in complex, high-dimensional control tasks, thereby highlighting the critical role of causal modeling in policy learning.

This work addresses the tendency of existing large language models to conflate textual associations or hallucinations with genuine causal evidence when applied to causal discovery, often lacking explicit grounding in data and assumptions. To remedy this, the authors propose a novel paradigm wherein AI agents assist—but do not autonomously generate—causal conclusions. This framework orchestrates data analysis, preprocessing, method recommendation, expert knowledge integration, and result interpretation to ensure that all causal inferences are rigorously based on empirical data, explicit assumptions, and formal algorithms. Built upon the causal-learn ecosystem, the team developed an online platform (causallearn.com) featuring agent-driven capabilities for data validation, context-aware retrieval, assumption articulation, and graphical model explanation. The efficacy and reliability of this human–AI collaborative approach are demonstrated through a case study on Big Five personality data.