Kandinsky Conformal Prediction: Beyond Class- and Covariate-Conditional Coverage

2025-02-24

AI Summary

Traditional conformal prediction provides marginal coverage guarantees but often exhibits uneven coverage across subpopulations. To address this, we propose Kandinsky Conformal Prediction—a novel framework introducing flexible overlapping group assignments and fractional group membership modeling. It extends conditional coverage guarantees from the covariate space $X$ to the joint $(X,Y)$ space, unifying and generalizing label-conditional, covariate-conditional, and Mondrian conformal prediction. Theoretically, we derive a minimax-optimal high-probability bound on conditional coverage. Methodologically, Kandinsky integrates fractional membership modeling, joint $(X,Y)$-based grouping, and robust statistical boundary analysis. Empirical evaluation across multiple real-world datasets demonstrates substantial improvements in subpopulation coverage equity while preserving prediction set compactness—achieving strict, fine-grained fairness-aware statistical guarantees across diverse subgroups.

Technology Category

• Machine Learning: Calibration & Uncertainty Quantification
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Conformal prediction is a powerful distribution-free framework for constructing prediction sets with coverage guarantees. Classical methods, such as split conformal prediction, provide marginal coverage, ensuring that the prediction set contains the label of a random test point with a target probability. However, these guarantees may not hold uniformly across different subpopulations, leading to disparities in coverage. Prior work has explored coverage guarantees conditioned on events related to the covariates and label of the test point. We present Kandinsky conformal prediction, a framework that significantly expands the scope of conditional coverage guarantees. In contrast to Mondrian conformal prediction, which restricts its coverage guarantees to disjoint groups -- reminiscent of the rigid, structured grids of Piet Mondrian's art -- our framework flexibly handles overlapping and fractional group memberships defined jointly on covariates and labels, reflecting the layered, intersecting forms in Wassily Kandinsky's compositions. Our algorithm unifies and extends existing methods, encompassing covariate-based group conditional, class conditional, and Mondrian conformal prediction as special cases, while achieving a minimax-optimal high-probability conditional coverage bound. Finally, we demonstrate the practicality of our approach through empirical evaluation on real-world datasets.

Problem

• Expands conditional coverage guarantees in conformal prediction
• Handles overlapping and fractional group memberships
• Achieves minimax-optimal high-probability conditional coverage

Innovation

• Flexible overlapping group memberships
• Unifies existing conformal prediction methods
• Achieves minimax-optimal coverage bounds