Adaptive Multi-Horizon Reinforcement Learning

2026-07-22

AI Summary

This work addresses the limitation of conventional reinforcement learning methods that employ a fixed discount factor, which often struggle to balance short-term and long-term rewards effectively in dynamic environments. To overcome this, the paper proposes an adaptive multi-horizon reinforcement learning approach that dynamically selects and fuses discounting strategies across multiple time scales. This enables the agent to automatically adapt to shifts in reward structures and task transitions without requiring manual hyperparameter tuning. Evaluated in continuous-task MiniGrid environments, the method demonstrates significantly improved parameter efficiency and environmental adaptability. It achieves this by online identification and composition of optimal discount factors, thereby enhancing agent performance in continual learning scenarios.

Abstract

Effective decision-making in complex and changing environments requires balancing short-term and long-term consequences. In reinforcement learning (RL), this trade-off is typically controlled through a fixed discount factor, which imposes a single exponentially discounted temporal horizon. However, biological agents exhibit flexible and adaptive temporal discounting, suggesting that effective planning requires multiple timescales. Here, we propose a multi-horizon approach that adaptively selects and combines temporal horizons, enabling robust adaptation to changes in reward structure without manual discount-factor tuning. This flexibility makes the method particularly suitable for continual learning scenarios involving task switches and varying environmental configurations. Empirically, we demonstrate that our approach identifies effective discount factors across a range of MiniGrid environments, including continual settings composed of three sequentially changing tasks. These results suggest that adaptive temporal discounting can improve parameter efficiency and enhance adaptability in both artificial and biologically inspired learning systems.

Problem

• reinforcement learning
• temporal discounting
• multi-horizon
• continual learning
• adaptive decision-making

Innovation

• adaptive discounting
• multi-horizon reinforcement learning
• continual learning
• temporal abstraction
• parameter efficiency