🤖 AI Summary
This work addresses the challenge of entropy coefficient selection in real-world reinforcement learning, where environmental non-stationarity often leads to insufficient or excessive exploration under a fixed entropy weight, compromising both efficiency during stable periods and adaptability following abrupt changes. To this end, the paper introduces Adaptive Entropy Scheduling (AES), the first method to frame entropy scheduling under non-stationary environments as a one-dimensional online trade-off problem. AES dynamically adjusts the temperature coefficient using a lightweight online proxy metric for distributional drift, without altering the underlying algorithm architecture. Evaluated across four algorithms, twelve tasks, and four distinct drift patterns, AES consistently reduces performance degradation and accelerates policy recovery after environmental shifts.
📝 Abstract
Real-world reinforcement learning often faces environment drift, but most existing methods rely on static entropy coefficients/target entropy, causing over-exploration during stable periods and under-exploration after drift (thus slow recovery), and leaving unanswered the principled question of how exploration intensity should scale with drift magnitude. We prove that entropy scheduling under non-stationarity can be reduced to a one-dimensional, round-by-round trade-off, faster tracking of the optimal solution after drift vs. avoiding gratuitous randomness when the environment is stable, so exploration strength can be driven by measurable online drift signals. Building on this, we propose AES (Adaptive Entropy Scheduling), which adaptively adjusts the entropy coefficient/temperature online using observable drift proxies during training, requiring almost no structural changes and incurring minimal overhead. Across 4 algorithm variants, 12 tasks, and 4 drift modes, AES significantly reduces the fraction of performance degradation caused by drift and accelerates recovery after abrupt changes.