Technology Category
Machine Learning: Reinforcement LearningReasoning under Uncertainty: Sequential Decision MakingMultiagent Systems: Multiagent Learning
From Pixels to Factors: Learning Independently Controllable State Variables for Reinforcement Learning
🤖 AI Summary
In reinforcement learning, factorized MDP algorithms achieve high sample efficiency but require prior knowledge of the state decomposition structure, whereas deep RL handles high-dimensional raw observations yet cannot exploit such structural priors. To bridge this gap, we propose Action-Controllable Factorization (ACF), the first method to self-supervisedly learn action-sparse, independently controllable latent variables directly from pixel inputs. ACF jointly leverages contrastive learning and action-triggered state-transition modeling, enforcing sparse responsiveness of latent factors to actions to achieve disentangled representation learning. Evaluated on Taxi, FourRooms, and MiniGrid-DoorKey benchmarks, ACF accurately recovers ground-truth controllable factors—outperforming existing disentanglement methods by a significant margin. Our approach establishes a new paradigm for leveraging inherent state structure in RL without requiring domain-specific structural priors.
Technology Category
Application Category
📝 Abstract
Algorithms that exploit factored Markov decision processes are far more sample-efficient than factor-agnostic methods, yet they assume a factored representation is known a priori -- a requirement that breaks down when the agent sees only high-dimensional observations. Conversely, deep reinforcement learning handles such inputs but cannot benefit from factored structure. We address this representation problem with Action-Controllable Factorization (ACF), a contrastive learning approach that uncovers independently controllable latent variables -- state components each action can influence separately. ACF leverages sparsity: actions typically affect only a subset of variables, while the rest evolve under the environment's dynamics, yielding informative data for contrastive training. ACF recovers the ground truth controllable factors directly from pixel observations on three benchmarks with known factored structure -- Taxi, FourRooms, and MiniGrid-DoorKey -- consistently outperforming baseline disentanglement algorithms.
Problem
Research questions and friction points this paper is trying to address.
Learning controllable latent variables from pixel observations
Bridging factored MDPs with deep reinforcement learning
Uncovering sparse action-variable relationships through contrastive learning
Innovation
Methods, ideas, or system contributions that make the work stand out.
Learns controllable state variables from pixels
Uses contrastive learning for factor discovery
Exploits action sparsity for improved disentanglement
🔎 Similar Papers
No similar papers found.
