🤖 AI Summary
This study addresses the poor out-of-distribution (OOD) generalization of reinforcement learning agents under changes in environment dynamics. To systematically evaluate rule-transfer generalization, the authors introduce a controllable benchmark based on one-dimensional cellular automata, where observation and action spaces remain fixed while only the transition rules are varied. The work proposes a standardized OOD evaluation protocol, statistical reporting guidelines, and an information-theoretic identity that characterizes the theoretical limits of uncertainty reduction. Empirical results demonstrate that algorithms exhibiting strong in-distribution performance suffer significant degradation under rule transfer, and that high-variance OOD evaluations require extensive repetition to ensure reliability. This work provides a reproducible evaluation framework alongside theoretical insights into generalization under dynamic rule shifts.
📝 Abstract
We present Tape, a controlled reinforcement-learning benchmark designed to isolate out-of-distribution (OOD) failure under latent rule shifts.Tape is derived from one-dimensional cellular automata, enabling precise train/test splits where observation and action spaces are held fixed while transition rules change. Using a reproducible evaluation pipeline, we compare model-free baselines, model-based planning with learned world models, and task-inference (meta-RL) methods. A consistent pattern emerges: methods that are strong in-distribution (ID) can collapse under heldout-rule OOD, and high-variance OOD evaluation can make rankings unstable unless experiments are sufficiently replicated.We provide (i) standardized OOD protocols, (ii) statistical reporting requirements (seeds, confidence intervals, and hypothesis tests), and (iii) information-theoretic identities connecting entropy reduction to conditional mutual information and expected posterior KL divergence, clarifying what"uncertainty reduction"objectives can and cannot guarantee under rule shifts.