🤖 AI Summary
To address the challenge of sustaining fairness auditing for deployed models under data policy drift, subgroup heterogeneity, and dynamic distributional shifts, this paper proposes the first dynamic fairness auditing framework tailored for streaming data. Methodologically, it innovatively adapts “testing by betting”—a game-theoretic statistical paradigm—into fairness monitoring, integrating anytime-valid inference with nonparametric, sample-size-agnostic sequential testing that adapts to time-varying sampling strategies. Unlike conventional batch-based approaches, our framework enables real-time, continuous fairness assessment without requiring prior knowledge of data distribution or fixed sample sizes. Experiments on three benchmark datasets demonstrate that the method achieves high detection sensitivity, low false positive rate, and strong robustness against concept drift and subgroup imbalance. These properties collectively enhance the practicality, timeliness, and deployability of fairness monitoring in real-world operational environments.
📝 Abstract
We provide practical, efficient, and nonparametric methods for auditing the fairness of deployed classification and regression models. Whereas previous work relies on a fixed-sample size, our methods are sequential and allow for the continuous monitoring of incoming data, making them highly amenable to tracking the fairness of real-world systems. We also allow the data to be collected by a probabilistic policy as opposed to sampled uniformly from the population. This enables auditing to be conducted on data gathered for another purpose. Moreover, this policy may change over time and different policies may be used on different subpopulations. Finally, our methods can handle distribution shift resulting from either changes to the model or changes in the underlying population. Our approach is based on recent progress in anytime-valid inference and game-theoretic statistics-the"testing by betting"framework in particular. These connections ensure that our methods are interpretable, fast, and easy to implement. We demonstrate the efficacy of our approach on three benchmark fairness datasets.
