Technology Category
Machine Learning: Causal LearningReasoning under Uncertainty: CausalitySearch and Optimization: Learning to Search
Causally Aligned Curriculum Learning
🤖 AI Summary
Reinforcement learning (RL) faces the “curse of dimensionality” in high-dimensional tasks, while conventional curriculum learning relies on the unrealistic assumption that source and target tasks share an invariant optimal policy. Method: This paper introduces “causal alignment” from a causal inference perspective to ensure transferability of optimal decision rules between source and target tasks. Based on causal graph models, we derive the first sufficient condition for causal alignment. Our framework integrates causal modeling, counterfactual reasoning, and an adaptive curriculum generation algorithm, supporting both discrete and continuous action spaces as well as pixel-level observations. Results: Experiments demonstrate that our approach significantly improves policy transfer efficiency and final performance under unobserved confounding—outperforming standard curriculum learning in robustness and convergence speed.
Technology Category
Application Category
📝 Abstract
A pervasive challenge in Reinforcement Learning (RL) is the"curse of dimensionality"which is the exponential growth in the state-action space when optimizing a high-dimensional target task. The framework of curriculum learning trains the agent in a curriculum composed of a sequence of related and more manageable source tasks. The expectation is that when some optimal decision rules are shared across source tasks and the target task, the agent could more quickly pick up the necessary skills to behave optimally in the environment, thus accelerating the learning process. However, this critical assumption of invariant optimal decision rules does not necessarily hold in many practical applications, specifically when the underlying environment contains unobserved confounders. This paper studies the problem of curriculum RL through causal lenses. We derive a sufficient graphical condition characterizing causally aligned source tasks, i.e., the invariance of optimal decision rules holds. We further develop an efficient algorithm to generate a causally aligned curriculum, provided with qualitative causal knowledge of the target task. Finally, we validate our proposed methodology through experiments in discrete and continuous confounded tasks with pixel observations.
Problem
Research questions and friction points this paper is trying to address.
Addresses curse of dimensionality in Reinforcement Learning
Ensures invariant optimal decision rules in curriculum
Generates causally aligned curriculum using causal knowledge
Innovation
Methods, ideas, or system contributions that make the work stand out.
Uses causal alignment for curriculum learning
Develops efficient causally aligned curriculum algorithm
Validates with experiments in confounded tasks
🔎 Similar Papers
2024-09-30arXiv.orgCitations: 0
