Audits Under Resource, Data, and Access Constraints: Scaling Laws For Less Discriminatory Alternatives

2025-09-06

AI Summary

In AI auditing, the “less-discriminatory alternative” (LDA) burden of proof poses a significant barrier for complainants, who must demonstrate the existence of a fairness-improved yet performance-equivalent alternative model—despite lacking access to the original model, training data, or computational resources. Method: We propose the first closed-form upper bound on the loss–fairness Pareto frontier, enabling low-resource extrapolation of fairness-performance trade-offs without training large models. Our approach leverages demographic parity and binary cross-entropy loss, fitting a context-specific frontier using only seven small models and minimal sample data. Contribution/Results: The resulting scaling law remains robust under non-ideal conditions—including data scarcity and black-box model constraints—substantially lowering the LDA verification threshold. Empirical evaluation confirms its effectiveness across diverse settings, offering a deployable technical foundation for anti-discrimination legal practice.

Technology Category

• Philosophy and Ethics of AI: Bias, Fairness & Equity
• Machine Learning: Ethics, Bias, and Fairness
• Natural Language Processing: Ethics — Bias, Fairness, Transparency & Privacy

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Economics, Online Markets and Human Computation: Fairness and ethical considerations in crowd work and in human-in-the-loop AI systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

AI audits play a critical role in AI accountability and safety. One branch of the law for which AI audits are particularly salient is anti-discrimination law. Several areas of anti-discrimination law implicate the "less discriminatory alternative" (LDA) requirement, in which a protocol (e.g., model) is defensible if no less discriminatory protocol that achieves comparable performance can be found with a reasonable amount of effort. Notably, the burden of proving an LDA exists typically falls on the claimant (the party alleging discrimination). This creates a significant hurdle in AI cases, as the claimant would seemingly need to train a less discriminatory yet high-performing model, a task requiring resources and expertise beyond most litigants. Moreover, developers often shield information about and access to their model and training data as trade secrets, making it difficult to reproduce a similar model from scratch. In this work, we present a procedure enabling claimants to determine if an LDA exists, even when they have limited compute, data, information, and model access. We focus on the setting in which fairness is given by demographic parity and performance by binary cross-entropy loss. As our main result, we provide a novel closed-form upper bound for the loss-fairness Pareto frontier (PF). We show how the claimant can use it to fit a PF in the "low-resource regime," then extrapolate the PF that applies to the (large) model being contested, all without training a single large model. The expression thus serves as a scaling law for loss-fairness PFs. To use this scaling law, the claimant would require a small subsample of the train/test data. Then, the claimant can fit the context-specific PF by training as few as 7 (small) models. We stress test our main result in simulations, finding that our scaling law holds even when the exact conditions of our theory do not.

Problem

• Addressing AI audit challenges under resource and data constraints
• Enabling detection of less discriminatory alternatives without full model access
• Providing scaling laws for fairness-performance trade-offs in low-resource settings

Innovation

• Closed-form upper bound for Pareto frontier
• Scaling law extrapolates without large training
• Requires only small data subsample and models