Task-Induced Representational Invariances Depend on Learning Objective in Deep RL

2026-06-01

AI Summary

This study addresses the weak theoretical foundation underlying learned representations in deep reinforcement learning, which hinders meaningful comparisons with animal learning mechanisms. Building on Markov decision process (MDP) reduction theory, the work systematically analyzes the representational structures acquired by algorithms such as DQN and PPO in navigation tasks. It reveals, for the first time, that value-based methods induce representations invariant under MDP homomorphic symmetries, whereas policy gradient methods yield representations invariant under action symmetries—a property linked to prompt dependence observed in large language models. Despite comparable task performance, these algorithms exhibit markedly different representational invariances, leading to divergent transfer capabilities. These findings offer a novel perspective on neural coding and brain-inspired learning.

Abstract

Reinforcement Learning (RL) has long served as a model for goal-directed animal behavior in neuroscience. Modern deep RL has shown remarkable success across many domains, further strengthening this connection. The ability to learn abstract representations of high-dimensional state spaces underlies much of this success. However, theoretical understanding of these learned representations remains limited, hindering direct comparisons between models and animal learning. We address this gap by analyzing deep RL representations through the lens of MDP reduction theory. Investigating canonical RL algorithms in a navigation task, we find that even when performance is comparable, the value-based method (DQN) learns representations that are invariant to MDP homomorphism symmetries, while the policy-gradient method (PPO) learns representations invariant to action symmetries. These differences emerge consistently across domains, have downstream consequences for transfer learning, and appear in LLMs in a prompt-dependent manner. Our findings provide a principled approach to comparing learned representations across RL algorithms, with demonstrated practical implications and possible insights for neural coding in the brain.

Problem

• representation
• deep reinforcement learning
• MDP homomorphism
• invariance
• neural coding

Innovation

• representational invariance
• MDP homomorphism
• deep reinforcement learning
• transfer learning
• neural coding