Ask-E: An Environment for Calibrated Question Generation

2026-08-07

AI Summary

This work proposes the Ask-E framework, designed to automatically generate high-quality questions that precisely align with a model’s capability boundaries, thereby training and evaluating large language models on question generation rather than answer accuracy. By defining the capability boundaries of two reference models, the method constructs a target difficulty interval and employs an automatic calibration mechanism combined with either reinforcement learning or supervised fine-tuning—enabling capability transfer without reliance on stronger models or correctness-based rewards. Experiments reveal that state-of-the-art models achieve below 50% accuracy on the calibrated tasks, underscoring the benchmark’s challenge. Models trained with Ask-E demonstrate significant improvements across multiple mathematical downstream tasks, despite using no additional data or external strong models.

Abstract

Today, we improve models by training and evaluating them on problems at the frontier of their abilities. Creating such problems is itself a demanding task, requiring the ability to probe model limits and generalize beyond existing question distributions. It also means placing problems at a precise difficulty level, which requires understanding what it takes to solve them. In short, generating problems calibrated to a model's current frontier demands capability beyond it, an increasingly burdensome constraint as models improve. Our key insight is that we can leverage this constraint to our advantage: a model that can generate problems consistently calibrated to a given frontier must possess capability beyond it. Accordingly, we present Ask-E, an environment that benchmarks and trains models on their ability to write questions at a given skill level, rather than answer them. Concretely, we define target skill levels as ranges bounded by the capabilities of two existing language models. A generated question is successfully calibrated if exactly one of the two models can solve it, placing it precisely within the target range and differentiating the capabilities of these models. Ask-E serves both as a benchmark and a training environment, where models generate problems calibrated to a variety of skill levels. We find that even frontier models achieve below 50% calibration on the benchmark, leaving significant headroom to measure future progress. We also show that training on this environment leads to improvements across a number of downstream math benchmarks even with no new math data, no interaction with stronger models, and no correctness-based reward.

Problem

• calibrated question generation
• model frontier
• skill-level calibration
• question benchmarking
• language model evaluation

Innovation

• calibrated question generation
• skill-level benchmarking
• model capability frontier
• self-improvement via problem generation
• Ask-E environment