CAOS: Conformal Aggregation of One-Shot Predictors

2026-01-08

AI Summary

This work proposes a novel conformal prediction framework to address the unreliable uncertainty quantification in single-sample prediction and the inefficiency of traditional conformal methods under data scarcity. By introducing a leave-one-out calibration strategy, the method adaptively aggregates multiple single-sample predictors, thereby circumventing the restrictive exchangeability assumption commonly required in conformal inference. Leveraging a monotonicity argument, it guarantees valid marginal coverage while enabling, for the first time, a principled conformal fusion of multiple single-sample predictors. Empirical evaluations on facial landmark localization and RAFT text classification benchmarks demonstrate that the proposed approach significantly reduces prediction set sizes while maintaining reliable coverage, outperforming existing baselines.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Reasoning under Uncertainty: Other Foundations of Reasoning under Uncertainty
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Security and Privacy: Large-scale security measurements

Abstract

One-shot prediction enables rapid adaptation of pretrained foundation models to new tasks using only one labeled example, but lacks principled uncertainty quantification. While conformal prediction provides finite-sample coverage guarantees, standard split conformal methods are inefficient in the one-shot setting due to data splitting and reliance on a single predictor. We propose Conformal Aggregation of One-Shot Predictors (CAOS), a conformal framework that adaptively aggregates multiple one-shot predictors and uses a leave-one-out calibration scheme to fully exploit scarce labeled data. Despite violating classical exchangeability assumptions, we prove that CAOS achieves valid marginal coverage using a monotonicity-based argument. Experiments on one-shot facial landmarking and RAFT text classification tasks show that CAOS produces substantially smaller prediction sets than split conformal baselines while maintaining reliable coverage.

Problem

• one-shot prediction
• conformal prediction
• uncertainty quantification
• data efficiency
• coverage guarantees

Innovation

• conformal prediction
• one-shot learning
• uncertainty quantification
• leave-one-out calibration
• prediction sets