Approximate Equivariance in Reinforcement Learning

2024-11-06

arXiv.org

AI Summary

Reinforcement learning (RL) often encounters approximate symmetries—structural invariances that hold only approximately—not exact symmetries—posing challenges for existing equivariant RL methods designed for strict symmetry. Method: This paper introduces the Approximate Equivariant Markov Decision Process (AE-MDP) framework, the first formal integration of approximate equivariance into RL theory and algorithm design. We propose relaxable group convolution and direction-aware convolutional networks to adaptively model symmetry deviations. Theoretically, we characterize how approximate equivariance affects the optimal Q-function and prove that our method achieves optimal performance under both exact and approximate symmetry. Results: Experiments on continuous control (SAC/PPO-based) and real-world stock trading demonstrate that our approach matches state-of-the-art equivariant methods under exact symmetry, significantly outperforms them under approximate symmetry, and improves robustness to test-time observation noise.

Technology Category

• Machine Learning: Adversarial Learning & Robustness
• Multiagent Systems: Adversarial Agents
• Game Theory and Economic Paradigms: Adversarial Learning

Application Category

• Economics, Online Markets and Human Computation: Uses of LLMs and GenAI for marketplace design, bidding, and strategic interactions
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

Equivariant neural networks have shown great success in reinforcement learning, improving sample efficiency and generalization when there is symmetry in the task. However, in many problems, only approximate symmetry is present, which makes imposing exact symmetry inappropriate. Recently, approximately equivariant networks have been proposed for supervised classification and modeling physical systems. In this work, we develop approximately equivariant algorithms in reinforcement learning (RL). We define approximately equivariant MDPs and theoretically characterize the effect of approximate equivariance on the optimal $Q$ function. We propose novel RL architectures using relaxed group and steerable convolutions and experiment on several continuous control domains and stock trading with real financial data. Our results demonstrate that the approximately equivariant network performs on par with exactly equivariant networks when exact symmetries are present, and outperforms them when the domains exhibit approximate symmetry. As an added byproduct of these techniques, we observe increased robustness to noise at test time. Our code is available at https://github.com/jypark0/approx_equiv_rl.

Problem

• Develops approximately equivariant algorithms for reinforcement learning
• Addresses tasks with approximate symmetry, not exact symmetry
• Improves robustness and performance in noisy environments

Innovation

• Approximately equivariant MDPs for RL
• Relaxed group convolutions in RL
• Steerable convolutions for improved robustness