🤖 AI Summary
This work addresses a critical limitation of existing conformal prediction methods, which guarantee only marginal coverage and fail to characterize key operational metrics—such as decision frequency, error exposure, and rejection rate—and their inherent trade-offs in real-world deployment. To overcome this, the authors propose an operational certification framework that goes beyond coverage by introducing a calibration-audit two-stage mechanism to quantify and guarantee the statistical properties of system behavior under finite-sample settings. Key innovations include Small-Sample Beta Correction (SSBC) for finite-sample coverage guarantees, reusable confidence envelopes for operational metrics, and the revelation of geometric couplings and trade-off boundaries among these metrics under conformal partitioning. The framework successfully generates auditable operational configuration menus on Tox21 and AquaSolDB benchmarks, explicitly delineating performance boundaries and uncertainties across different calibration strategies.
📝 Abstract
Deployed conformal predictors are long-lived decision infrastructure operating over finite operational windows. The real-world question is not only ``Does the true label lie in the prediction set at the target rate?''(marginal coverage), but ``How often does the system commit versus defer? What error exposure does it induce when it acts? How do these rates trade off?''Marginal coverage does not determine these deployment-facing quantities: the same calibrated thresholds can yield different operational profiles depending on score geometry. We provide a framework for operational certification and planning beyond coverage with three contributions. (1) Small-Sample Beta Correction (SSBC): we invert the exact finite-sample Beta/rank law for split conformal to map a user request $(\alpha^\star,\delta)$ to a calibrated grid point with PAC-style semantics, yielding explicit finite-window coverage guarantees. (2) Calibrate-and-Audit: since no distribution-free pivot exists for rates beyond coverage, we introduce a two-stage design in which an independent audit set produces a reusable region -- label table and certified finite-window envelopes (Binomial/Beta-Binomial) for operational quantities -- commitment frequency, deferral, decisive error exposure, and commit purity -- via linear projection. (3) Geometric characterization: we describe feasibility constraints, regime boundaries (hedging vs.\ rejection), and cost-coherence conditions induced by a fixed conformal partition, explaining why operational rates are coupled and how calibration navigates their trade-offs. The output is an auditable operational menu: for a fixed scoring model, we trace attainable operational profiles across calibration settings and attach finite-window uncertainty envelopes. We demonstrate the approach on Tox21 toxicity prediction (12 endpoints) and aqueous solubility screening using AquaSolDB.