🤖 AI Summary
This study addresses the challenge of efficiently comparing and selecting among multiple adaptive prediction pipelines under hard coverage constraints. To this end, it proposes the CC-SMCS framework, which achieves sequential model selection by decoupling feasibility from optimality. Technically, the approach employs stochastic constrained argmin modeling, simultaneous martingale confidence sequences, and rectangular region projections, yielding closed-form rules with finite-sample guarantees without requiring stationarity assumptions. The proposed method contains all constrained optimal pipelines with high probability while supporting data-dependent stopping and delayed feedback scenarios. Furthermore, this work establishes an impossibility result for margin-safe certification, thereby providing a theoretical foundation for constrained online learning.
📝 Abstract
Modern conformal forecasting systems often maintain several adaptive pipelines that differ in base forecasters, conformity scores, calibration windows, and update rules. Comparing them is difficult because coverage is a hard constraint, whereas efficiency should be optimized only among feasible pipelines. We formulate this problem as sequential inference for a stochastic constrained argmin. At each time, the target is the set of minimum-cost pipelines satisfying multiple prefix-average conditional miscoverage constraints. We introduce Coverage-Constrained Sequential Model Confidence Sets (CC-SMCS), which separate certifiably feasible, possibly feasible, and possibly constrained-optimal pipelines. Using simultaneous martingale confidence sequences, CC-SMCS projects a rectangular confidence region onto the constrained argmin and admits an exact closed-form rule. With probability at least $1-δ$, it contains every constrained-optimal pipeline simultaneously over all times. This finite-sample guarantee requires no stationarity or mixing assumptions and remains valid under data-dependent stopping. We also establish an impossibility result for safe certification at the coverage boundary and extend the construction to delayed multi-horizon feedback and outcome-dependent efficiency objectives.