Task diversity produces systematic transfer but inhibits continual reinforcement learning

2026-05-30

AI Summary

The impact of task diversity on out-of-distribution adaptation in continual reinforcement learning remains unclear. This work proposes Banyan—a GPU-accelerated continual reinforcement learning environment—that, for the first time, disentangles task diversity into three independently controllable dimensions: map layout, interactive objects, and subgoal dependencies, enabling a systematic investigation of their influence on transfer and continual learning. Experiments reveal that high diversity substantially improves zero-shot transfer performance, allowing agents to achieve initial performance on new tasks nearly matching that on previously seen tasks. However, it concurrently hinders long-term learning: as the number of distribution shifts increases, agent performance plateaus and forgetting of prior tasks intensifies. These findings uncover a fundamental trade-off between transferability and continual learning efficacy induced by task diversity, offering a new perspective for designing efficient continual learning algorithms.

Abstract

Continual reinforcement learning aims to produce agents that learn not only to improve at their current tasks but also to adapt as task distributions change. Training an agent on many diverse tasks can induce zero-shot generalization, but previous work generally evaluates this generalization after training -- with frozen weights. Whether task diversity also improves an agent's ability to continue learning across distribution shifts remains unclear. We introduce Banyan, a GPU-accelerated continual RL domain in which task diversity factors into three independently controllable axes: the map layouts an agent must navigate, the objects it must interact with, and the hierarchical structures of sub-goal dependencies. Across individual distribution shifts, increasing diversity along each axis causes agents to begin training on the new tasks near the performance attained on the previous one, even when the shift changes the structure of the optimal policy. However, as the number of shifts increases, this local transfer does not by itself yield sustained continual learning: longer-horizon tasks plateau, and earlier task distributions are forgotten after later training. Banyan is a benchmark for studying when controlled task diversity produces transferable learning, when that transfer persists, and where it falls short of proper continual learning.

Problem

• continual reinforcement learning
• task diversity
• distribution shift
• transfer learning
• catastrophic forgetting

Innovation

• continual reinforcement learning
• task diversity
• zero-shot generalization
• distribution shift
• Banyan benchmark