On the Runtime Analysis of Reinforcement Learning Hyper-Heuristics

2026-07-24

AI Summary

This work addresses the long-standing challenge that reinforcement learning-based hyper-heuristics (RLHH) often struggle to effectively select low-level heuristics on standard benchmark functions. Focusing on the LeadingOnes problem and combining two randomized local search operators—RLS₁ and RLS₂—the study provides the first rigorous theoretical proof, supported by empirical validation, that RLHH can achieve the optimal expected runtime attainable by these two operators (up to lower-order terms) under appropriate parameter settings. This result refutes prior skepticism regarding RLHH’s capacity to learn effective heuristic selection policies. Furthermore, experimental results demonstrate that, at practical problem scales, RLHH outperforms the generalized randomized gradient hyper-heuristic, which also enjoys optimal theoretical runtime guarantees.

Abstract

Selection Hyper-heuristics (HHs) automate algorithmic design by selecting from a set of low-level heuristics which one to apply at each stage of the optimisation process. Several impressive results have been recently rigorously proven regarding the performance of selection hyper-heuristics (HHs) for standard benchmark functions. However, the learning mechanisms employed by these HHs are considerably simplified compared to the machine learning techniques typically used in real world applications. In this paper we analyse a Reinforcement Learning Hyper-heuristic (RLHH) from the literature. The only previous result available proved that for a wide range of parameter settings, RLHH does not learn to select heuristics appropriately for the standard LeadingOnes benchmark function. In this paper, we rigorously prove that with appropriate parameter values RLHH equipped with two random local search operators, RLS_1 and RLS_2 optimises the LeadingOnes benchmark function in the best possible expected runtime achievable with the two operators up to lower order terms. Experiments show that for realistic problem sizes it is faster than the Generalised Random Gradient HH which was previously proven to also have optimal expected runtime up to lower order terms.

Problem

• Reinforcement Learning Hyper-heuristics
• Runtime Analysis
• Selection Hyper-heuristics
• LeadingOnes
• Algorithm Selection

Innovation

• Reinforcement Learning Hyper-heuristic
• Runtime Analysis
• LeadingOnes
• Optimal Expected Runtime
• Selection Hyper-heuristics