BayesAME: Bayesian Active Model Evaluation

2026-07-29

AI Summary

This work addresses the high cost of evaluating large language models on benchmark suites by proposing a sequential Bayesian framework that efficiently estimates overall performance using a small coreset of test instances. The method introduces latent competence variables to model a model’s historical performance, leverages Bayesian posterior inference to quantify both performance and uncertainty, and iteratively expands the coreset based on an information gain criterion until predefined accuracy and uncertainty thresholds are met. Key innovations include the first automatic determination of coreset size to ensure reliable evaluation, explicit modeling of inter-model performance correlations through multi-task dependencies, and the use of continuous log-likelihood to enhance estimation precision. Experiments demonstrate that the approach significantly outperforms existing sequential strategies, achieving high-fidelity assessment with substantially smaller coresets and validating the efficacy of non-random sampling and continuous response modeling.

Abstract

Evaluating large generative models across benchmarks is time-consuming and computationally expensive. This drives the need for methods that can estimate full benchmark performance by evaluating models on only a subset of items, known as a coreset. Current literature mostly requires the practitioner to input a coreset size. However, when reliable performance estimation takes priority over efficiency, an evaluation method should also be capable of automatically determining a coreset size that reflects this priority. We introduce BayesAME, a sequential Bayesian framework specifically targeting automatic determination of the coreset size. BayesAME models performance as a random variable by defining a latent ability for each group of items sharing the same historical model performances, with a joint prior distribution encoding the belief that the target model behaves similarly to these historical models. The posterior distribution over these abilities is used to derive performance estimators, quantify performance uncertainty, and select items to add to the coreset via an information-gain criterion. The coreset is iteratively augmented until the performance estimate fluctuation and the performance uncertainty fall below their respective user-defined thresholds. We propose a multi-target extension that captures performance correlations across multiple target models to further reduce the coreset size. Through extensive experiments across diverse benchmarks, we demonstrate that BayesAME consistently outperforms sequential adaptations of existing methods. Crucially, our comprehensive analysis addresses recent skepticism in the literature, establishing that non-random coreset selection is advantageous over random selection. Finally, we highlight that leveraging continuous response log-likelihoods over traditional binary scores significantly enhances estimation accuracy.

Problem

• Bayesian active evaluation
• coreset selection
• generative model benchmarking
• performance estimation
• automatic coreset sizing

Innovation

• Bayesian active evaluation
• coreset selection
• performance estimation
• information gain
• multi-target modeling