Efficient Sequential Evaluation of Large Language Models

2026-07-19

AI Summary

This study addresses the problem of efficiently and sequentially evaluating the capabilities of new large language models on a fixed set of questions while rapidly narrowing the confidence interval around their performance. To this end, the authors propose a confidence sequence construction method based on Reverse Information Projection (RIPr) combined with betting-based testing. They theoretically establish the optimality of RIPr under idealized assumptions and identify key factors governing the rate of interval contraction. Building on this foundation, they design a hybrid active querying strategy that balances growth-oriented exploration, prediction-driven optimization, and uniform sampling. Experimental results across multiple synthetic benchmarks demonstrate that the proposed approach effectively characterizes confidence interval shrinkage dynamics and reveal that uniform sampling can outperform more sophisticated adaptive strategies in certain scenarios.

Abstract

We study the problem of sequentially evaluating a new large language model (LLM) on a fixed question set using historical performance data from prior LLMs. Our goal is to construct a confidence sequence (CS) for the model's capability on this question set and to design active querying rules that shrink the CS width as quickly as possible. For CS construction, we invert a family of test supermartingales and focus on two representative approaches: a reverse information projection (RIPr)-based approach and a testing-by-betting-based approach. We first study these approaches under an oracle setting, and demonstrate the oracle optimality of the RIPr-based construction. We then propose a growth-oriented querying rule that aims to maximize the worst-case one-step expected log-increment over the endpoints of the current CS. In practice, we build these test supermartingales and the querying rule on predictions of question-level correctness learned from historical data. We then analyze the shrinkage behavior of the resulting CSs and identify two key factors that slow the shrinkage rate of CSs: accumulated prediction mismatch and the spikiness of the querying distribution. Finally, motivated by this analysis, we propose several mixture querying rules that combine growth-oriented querying, prediction refinement, and uniform exploration, trying to mitigate the effects that slow the shrinkage rate. We provide experiments comparing different querying rules for the RIPr-based and testing-by-betting-based CSs across several synthetic testing datasets. Interestingly, we observe that the simplest querying rule, uniform sampling, can sometimes outperform more adaptive querying rules for both methods.

Problem

• sequential evaluation
• large language models
• confidence sequences
• active querying
• historical performance data

Innovation

• confidence sequence
• sequential evaluation
• test supermartingale
• active querying
• large language models