Which LLM to pick? Online Active Model Selection for Large Language Models

2026-10-01

AI Summary

This study addresses the challenges of benchmark misalignment and costly human annotation in the streaming deployment of large language models (LLMs) by proposing the ONLINE LLM PICKER framework. As the first active model selection method tailored for online LLM scenarios, this framework integrates active learning with online algorithms to adaptively identify the most informative prompts for annotation, thereby efficiently determining the optimal candidate model under a limited budget. Experimental results demonstrate that the proposed approach reduces annotation costs by 71.67% while accurately selecting the best-performing model, yielding a 2.51-fold decrease in cumulative generative regret. Overall, this work achieves low-cost, highly reliable online LLM decision-making.

Abstract

Large Language Models (LLMs) are increasingly applied to process streaming data, with practitioners relying on benchmarks to select the best model even though these signals only approximate real performance. While oracle annotations can provide reliable feedback, they are often costly and difficult to obtain at scale. To address this challenge, we propose ONLINE LLM PICKER, the first framework for active model selection for LLMs in online settings. Given an arbitrary stream of queries and a limited annotation budget, ONLINE LLM PICKER selects the most informative prompts for annotation to identify the best LLM among candidate models. Across multiple tasks including 10 datasets, for over 130 language models, we show that ONLINE LLM PICKER saves annotation cost by up to 71.67% while reliably identifying the best or near-best model for the stream. We also show that using the returned model for sequential generation on unannotated prompts across the stream reduces regret by up to a factor of 2.51x, indicating that ONLINE LLM PICKER can identify the best or near-best model well before processing all streaming prompts.

Problem

• Large Language Models
• Active Model Selection
• Online Learning
• Annotation Budget
• Streaming Data

Innovation

• Active Model Selection
• Large Language Models
• Online Learning
• Annotation Budget
• Regret Minimization