Thermodynamics of Reinforcement Learning Curricula

2026-03-12

AI Summary

This work investigates the design of optimal curriculum strategies to enhance the efficiency and generalization of reinforcement learning. Inspired by nonequilibrium thermodynamics, it treats task reward parameters as coordinates on a task manifold and establishes, for the first time, a theoretical connection between curriculum learning and thermodynamic work: the optimal curriculum corresponds to a geodesic path in task space that minimizes excess work. Building upon this geometric principle, the paper proposes the Minimum Excess Work (MEW) algorithm, which integrates differential geometry, maximum-entropy reinforcement learning, and temperature annealing to automatically generate curricula with provable optimality in terms of temperature schedules. This approach provides a principled and computationally tractable geometric framework for curriculum generation.

Technology Category

• Search and Optimization: Learning to Search
• Machine Learning: Learning with Manifolds
• Intelligent Robots: Learning & Optimization for ROB

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Connections between statistical mechanics and machine learning have repeatedly proven fruitful, providing insight into optimization, generalization, and representation learning. In this work, we follow this tradition by leveraging results from non-equilibrium thermodynamics to formalize curriculum learning in reinforcement learning (RL). In particular, we propose a geometric framework for RL by interpreting reward parameters as coordinates on a task manifold. We show that, by minimizing the excess thermodynamic work, optimal curricula correspond to geodesics in this task space. As an application of this framework, we provide an algorithm, "MEW" (Minimum Excess Work), to derive a principled schedule for temperature annealing in maximum-entropy RL.

Problem

• curriculum learning
• reinforcement learning
• task scheduling
• optimal curricula
• thermodynamics

Innovation

• non-equilibrium thermodynamics
• curriculum learning
• task manifold
• geodesic
• maximum-entropy reinforcement learning