PACE: Parameter Change for Unsupervised Environment Design

2026-05-02

AI Summary

This work addresses a critical limitation in existing unsupervised environment design (UED) methods, which rely on biased, high-variance, or computationally expensive proxy metrics that poorly reflect an agent’s true learning progress. To overcome this, the authors propose PACE, a novel approach that introduces the squared L2 norm of policy parameter updates as a direct measure of environmental value. By leveraging a first-order approximation, PACE efficiently and accurately quantifies an environment’s contribution to policy optimization without requiring additional environment interactions, yielding low-variance evaluations. Empirical results demonstrate that PACE significantly outperforms prior UED methods on MiniGrid and Craftax benchmarks, achieving an interquartile mean (IQM) score of 96.4% in out-of-domain MiniGrid evaluation and reducing the optimality gap to 17.2%.

Abstract

Unsupervised Environment Design (UED) offers a promising paradigm for improving reinforcement learning generalization by adaptively shaping training environments, but it requires reliable environment evaluation to remain effective. However, existing UED methods evaluate environments using indirect proxy signals such as regret, value-based errors, or Monte Carlo, which suffer from bias, high variance, or substantial computational overhead and fail to reflect agent realized learning progress. To address these limitations, we propose Parameter Change Environment Design (PACE), which evaluates an environment through the policy parameter change induced by training on that environment, directly grounding environment selection in realized learning progress. Specifically, PACE assigns environment value using a first-order approximation of the policy optimization objective, where the improvement induced by an environment is proportional to the squared L2 norm of the corresponding parameter update, enabling low-variance and computation-efficient evaluation without additional rollouts. Experiments on MiniGrid and Craftax show that PACE consistently outperforms established UED baselines, achieving higher IQM and smaller Optimality Gap on OOD evaluations, including an IQM of 96.4% and an Optimality Gap of 17.2% on MiniGrid.

Problem

• Unsupervised Environment Design
• environment evaluation
• reinforcement learning generalization
• learning progress
• policy optimization

Innovation

• Unsupervised Environment Design
• Parameter Change
• Policy Optimization
• Generalization in RL
• Environment Evaluation