Top-k on a Budget: Adaptive Ranking with Weak and Strong Oracles

2026-01-28

AI Summary

This work addresses the challenge of efficiently identifying the top-k best items in settings where strong evaluations are costly. The authors propose ACE and ACE-W algorithms that synergistically combine abundant but noisy weak oracles with scarce yet accurate strong oracles. By adaptively focusing verification on boundary items and dynamically allocating the weak oracle budget, the approach employs a two-stage screening-and-verification framework grounded in confidence interval analysis and incorporates an approximate threshold quality model. Theoretically, the number of strong oracle calls is shown to be O(m·ε_max⁴), matching the Ω(m·ε_max) lower bound up to polynomial factors. Experimental results demonstrate that the proposed methods substantially reduce the cost of strong queries while maintaining high identification accuracy.

Technology Category

• Search and Optimization: Algorithm Configuration
• Constraint Satisfaction and Optimization: Constraint Learning and Acquisition
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Identifying the top-$k$ items is fundamental but often prohibitive when exact valuations are expensive. We study a two-oracle setting with a fast, noisy weak oracle and a scarce, high-fidelity strong oracle (e.g., human expert verification or expensive simulation). We first analyze a simple screen-then-certify baseline (STC) and prove it makes at most $m(4\varepsilon_{\max})$ strong calls given jointly valid weak confidence intervals with maximum radius $\varepsilon_{\max}$, where $m(\cdot)$ denotes the near-tie mass around the top-$k$ threshold. We establish a conditional lower bound of $\Omega(m(\varepsilon_{\max}))$ for any algorithm given the same weak uncertainty. Our main contribution is ACE, an adaptive certification algorithm that focuses strong queries on critical boundary items, achieving the same $O(m(4\varepsilon_{\max}))$ bound while reducing strong calls in practice. We then introduce ACE-W, a fully adaptive two-phase method that allocates weak budget adaptively before running ACE, further reducing strong costs.

Problem

• top-k selection
• adaptive ranking
• weak oracle
• strong oracle
• budget constraint

Innovation

• adaptive ranking
• top-k selection
• weak and strong oracles
• budget-constrained learning
• active certification