Frontier Learning: Training LLM Reasoners at the Edge of Capability

2026-09-28

AI Summary

This study addresses the rapid depletion of effective training signals during reinforcement learning (RL) post-training of large language models (LLMs) caused by fixed question pools, which limits sustained improvements in reasoning capabilities. To overcome this, we propose a frontier exploration method that treats program generator parameters as a search space and leverages regret signals to prioritize exploring model capability boundaries, thereby enabling online, dynamic, and adaptive generation of training problems. This approach is integrated with the GRPO loss for open-ended RL post-training. Experiments demonstrate that our method significantly outperforms fixed-pool baselines across multiple reasoning tasks and model families, validating the effectiveness of dynamic data generation in continuously unlocking the reasoning potential of LLMs.

Abstract

Reinforcement Learning-based post-training of Large Language Models (LLM) has been successfully applied to improve their reasoning capabilities. Existing pipelines primarily finetune LLMs on a fixed pool of problems specified prior to training using the GRPO loss. This is fundamentally limiting, as learning signal arises only when policy rollouts mix successes and failures, causing the useful portion of any fixed pool to quickly become stale as the model improves. To address this, we propose frontier learning, an open-ended post-training approach in which procedural generators are used online to continually produce informative training problems. It treats the generator's task-specific parameters as a search space and uses a regret signal to prioritize and explore frontier difficulty levels in order to focus training at the edge of the model's evolving reasoning capabilities. Across several reasoning tasks and model families, our approach consistently achieves higher relative gains over fixed-pool baselines, demonstrating that effective post-training requires not only selecting useful problems, but continually generating them at the edge of capability.

Problem

• Large Language Models
• Reinforcement Learning
• Post-training
• Reasoning
• Fixed Problem Pool

Innovation

• Frontier Learning
• Reinforcement Learning
• Procedural Generation
• Open-ended Training
• LLM Reasoning