Scalable Causal Imitation Learning

2026-07-18

AI Summary

This work addresses the significant performance degradation of conventional imitation learning in long-horizon, high-dimensional continuous control tasks when unobserved confounders are present and the observations available to the expert and the imitator differ. To tackle this challenge, the authors propose two off-policy causal imitation learning algorithms—Causal SQIL and Causal IQ-Learn—that, for the first time, integrate causal inference—based on an efficient approximation of the sequential π-backdoor criterion—with soft Q-learning and inverse soft Q imitation learning. A fixed-size sliding window mechanism is introduced to reduce the computational complexity of causal adjustment. Empirical results demonstrate that the proposed methods substantially outperform existing causal imitation approaches across various confounded environments, even surpassing expert performance in long-horizon tasks, whereas methods ignoring causal structure fail completely.

Abstract

Imitation learning enables learning a policy in an unknown environment with a latent reward signal using expert demonstrations, but it struggles when the imitator's and expert's observations are mismatched and unobserved confounders are present in expert demonstrations. By identifying appropriate adjustment sets via the sequential $π$-backdoor criterion, causal imitation learning (CIL) provides a framework for approximating the expert's policy from confounded data. However, existing CIL methods, Causal Behavioral Cloning (Causal BC) and Causal Generative Adversarial Imitation Learning (Causal GAIL), are designed for short-horizon, low-dimensional settings. When applied to continuous control tasks with long horizons and high-dimensional state-action spaces, these methods exhibit poor performance: Causal BC suffers from compounding errors, Causal GAIL is unstable and sample-inefficient, and sequential $π$-backdoor adjustment becomes impractical. We introduce Causal Soft Q Imitation Learning (SQIL) and Causal Inverse soft-Q Learning (IQ-Learn), two off-policy causal imitation learning algorithms that combine the causal adjustment framework with state-of-the-art inverse reinforcement learning objectives. Both algorithms operate on causally-adjusted state representations produced by an efficient approximation of the sequential $π$-backdoor criterion, exploiting the causal structure of continuous control environments to reduce the full-horizon adjustment to a fixed-size sliding window. We evaluate all methods in a suite of confounded environments and find that Causal SQIL and Causal IQ-Learn substantially outperform prior CIL algorithms on long-horizon tasks, sometimes surpassing the expert, whereas all causally unaware imitation methods fail to learn meaningful behavior.

Problem

• causal imitation learning
• unobserved confounders
• observation mismatch
• long-horizon tasks
• high-dimensional control

Innovation

• Causal Imitation Learning
• Sequential π-backdoor
• Soft Q Imitation Learning
• Inverse Reinforcement Learning
• Off-policy Learning