precision-recall evaluation

Large-scale spatial and entity matching suffers from imprecise recall control (bias often exceeding several percentage points), high variance, and substantial manual verification costs. Method: This paper proposes an end-to-end calibration framework integrating a multi-threshold estimator, decile-based score stratification, inverse-variance-weighted subsample ensembling, alongside equi-grid bounding-box filtering, CSR-based candidate representation, deterministic xxHash bootstrapping, and a lightweight neural ranker. Contribution/Results: It achieves, for the first time under highly skewed score distributions, sub-percentage-point (<1%) variance in recall control while precisely attaining target recall rates. Evaluated on real cadastral datasets of 6.31M and 67.34M pairs, the method significantly reduces redundant verification, enables full pipeline deployment on a single TPU v3 core, and ensures reproducibility, low variance, and hardware efficiency.

Evaluating generative models via scalar metrics fails to characterize the trade-off between sample fidelity and diversity, particularly for long-range dependencies. To address the lack of theoretical foundations and a unified framework for precision–recall (PR) curve estimation, this work introduces a novel paradigm grounded in binary classification theory. Our framework derives a minimax upper bound on the PR estimation risk, unifying and generalizing classical metrics—including FID, precision, and recall—within a coherent statistical framework. Leveraging nonparametric estimation and rigorous statistical risk analysis, we establish the statistical optimality of our estimators. Extensive experiments across diverse generative modeling scenarios reveal heterogeneous PR curve behaviors, demonstrating significant improvements in evaluation granularity, interpretability, and reliability.

Current vector database evaluation over-relies on mean recall, obscuring inter-query performance variance and undermining robustness on hard queries—thereby compromising reliability in downstream tasks like RAG. This work advocates shifting from “average performance” to a “distributional robustness” evaluation paradigm. We propose Robustness-δ@K, a novel metric quantifying performance consistency in the tail of the recall distribution (e.g., the worst 10% of queries). We further design a thresholded query coverage metric and integrate it into mainstream benchmarks (BEIR, MTEB) to re-evaluate prominent indexes—including HNSW, IVF, and DiskANN. Experiments show that, at identical mean recall, indexes with higher Robustness-δ@K yield up to 12.3% improvement in RAG answer accuracy. Moreover, we identify graph connectivity and cluster balance as critical architectural factors governing retrieval robustness.

This paper challenges the widely held assumption in machine learning that the Area Under the Precision-Recall Curve (AUPRC) is universally superior to the Area Under the ROC Curve (AUROC) under class imbalance—and investigates its implications for algorithmic fairness. Method: We conduct rigorous theoretical analysis, experiments on semi-synthetic and real-world fairness-sensitive datasets, and a large-scale bibliometric study covering over one million publications. Contribution/Results: We provide the first formal proof that AUPRC is not generally advantageous under extreme imbalance; instead, it systematically amplifies group-level bias by favoring subpopulations with higher positive-class density. We trace and empirically refute the long-standing misconception that “AUPRC is inherently better.” Furthermore, we establish verifiable criteria delineating the applicability boundaries of AUROC versus AUPRC, and propose a principled, imbalance- and subgroup-aware framework for metric selection—thereby offering both theoretical foundations and practical guidance for fair model evaluation.

Current evaluations of large language models predominantly rely on static, clean benchmarks that fail to capture model robustness against real-world user inputs containing noise, typos, or semantically equivalent yet lexically diverse prompts. This work proposes a theoretical framework that decomposes model performance variance into contributions from data difficulty and prompt variation, and introduces BrittleBench—an evaluation pipeline that systematically assesses robustness by generating prompt variants through semantics-preserving textual perturbations. Experiments reveal that such perturbations can degrade model performance by up to 12%, alter model rankings in 63% of cases, and account for as much as 50% of performance variance in certain models. These findings demonstrate that state-of-the-art models are highly sensitive to minor prompt alterations, underscoring the critical need for dynamic, robustness-oriented evaluation protocols.

Current evaluation practices for supervised learning models are often misleading due to an overreliance on single aggregate metrics, which neglect the alignment among data characteristics, task objectives, and real-world application contexts. This work reframes model evaluation as a context-dependent, decision-oriented process and systematically investigates—through controlled experiments—the impact of dataset properties, validation strategies, class imbalance, and asymmetric error costs on evaluation outcomes. Leveraging diverse benchmark datasets, multiple validation protocols, and multidimensional performance measures, the study uncovers common pitfalls such as the accuracy paradox, data leakage, and metric misuse. It proposes a structured evaluation framework explicitly aligned with operational goals, offering principled guidance for developing more robust, reliable, and trustworthy supervised learning systems.

This work addresses the susceptibility of existing calibration evaluations for large language models to accuracy disparities, which distorts cross-model comparisons. To enable fair calibration assessment while controlling for accuracy, the authors propose the ACE framework, incorporating three alignment mechanisms: instance alignment, distribution alignment, and candidate alignment. The study systematically reveals, for the first time, the bias inherent in conventional global calibration metrics—such as Expected Calibration Error and Brier Score—when used for comparing models with differing accuracies, and introduces an accuracy-controlled correction strategy. Experimental results demonstrate that the apparent calibration advantages of most models substantially diminish—and their rankings frequently reverse—once calibration metrics are adjusted for accuracy, thereby demonstrating that unadjusted metrics are unsuitable for cross-model calibration evaluation.

Current performance evaluation metrics—such as accuracy and F1 score—are typically reported as point estimates, ignoring the uncertainty induced by data clustering structures. This oversight often leads to underestimation of variability and potentially misleading model comparisons. To address this, this work proposes a unified framework that expresses a broad class of performance metrics as smooth functionals of the confusion matrix probabilities. By integrating a cluster-robust sandwich variance estimator, the framework enables valid confidence interval construction, hypothesis testing, and paired model comparison. It represents the first systematic application of cluster-robust inference to predictive performance evaluation, accommodating both binary and multiclass settings, and further provides asymptotic theory–based methods for power and sample size calculations. Simulations demonstrate that the proposed approach achieves near-nominal coverage across diverse dependence structures and substantially outperforms conventional methods that ignore clustering; real-data analyses confirm that accounting for clustering can materially alter evaluation conclusions.

This work addresses the challenge that existing model evaluation methods often fail to reliably assess estimator quality in low-variance settings due to confounding between bias and variance or excessive sensitivity of statistical tests. To overcome this limitation, the authors propose a fault-tolerant evaluation framework that unifies bias and variance modeling through an adjustable tolerance parameter ε, enabling robust assessment of sample-efficient performance estimators within practically acceptable error margins. The framework integrates bias-variance analysis, fault-tolerant evaluation theory, and an adaptive ε-optimization algorithm, making it particularly well-suited for scenarios with low annotation costs. Experimental results demonstrate that the proposed approach provides a more comprehensive and reliable characterization of estimator behavior, significantly enhancing both the practical utility and stability of performance evaluation.

This work addresses the challenge of certifying the safety of large language models (LLMs) by statistically verifying that their failure rate lies below a specified threshold. A key obstacle is that using LLMs as evaluators introduces noise and bias, compromising statistical validity. To overcome this, the authors propose a “noisy-but-valid” hypothesis testing framework that estimates the evaluator’s true positive rate (TPR) and false positive rate (FPR) via a small human-annotated calibration set, then adjusts the statistical decision threshold accordingly to rigorously control Type I error with limited samples. This approach is the first to systematically handle imperfect evaluators with theoretical guarantees on when noisy evaluation outperforms direct assessment. It introduces the “Oracle Gap” to quantify the cost of parameter estimation and reveals how evaluation efficacy depends on evaluator quality, data scale, and certification requirements. Experiments on Jigsaw Comment, Hate Speech, and SafeRLHF datasets demonstrate superior performance over existing black-box estimators and enable interpretable reliability diagnostics.