Fairness Theatre: Evaluating Post-Hoc Fairness Interventions in Vendor-Controlled Early Warning Systems

2026-09-29

AI Summary

This study evaluates the practical efficacy of ex post fairness interventions within vendor-controlled early warning systems under procurement constraints. Leveraging student record data to construct simulated procurement scenarios, we systematically compare six intervention strategies and track their error distributions. We introduce the concept of "fairness theater" alongside an error-type profiling framework, revealing a fairness illusion wherein metric convergence fails to alleviate disparate group burdens. Our results demonstrate that existing interventions merely redistribute rather than eliminate disparities; notably, certain approaches exacerbate the conditions of marginalized groups due to their dependence on group size. These findings provide critical caveats for algorithmic fairness practices in real-world deployment contexts.

Abstract

Public institutions increasingly procure AI systems whose design they cannot inspect or change. In higher education, proprietary Early Warning Systems (EWS) leave colleges with few options beyond adjusting model outputs to address inequity. This raises the question of how fairness work is coordinated among vendors, institutions, advisors, and students with unequal power to change these systems? Using student records from a public college in Ontario, Canada, we evaluate six post-hoc fairness interventions on a research EWS under simulated procurement constraints. We compare fairness, accuracy, and demographic disparities, introducing error-type profiling to trace how interventions redistribute false positives and false negatives. Interventions redistributed disparities without consistently reducing them. Two implementations favored already-advantaged groups because they used group size to define disadvantage; small, marginalized groups remained poorly served. These findings show how procurement constraints and implementation choices shape the possibilities for fairness work. We call the resulting condition fairness theatre; dashboard metrics converge while groups' error burdens persist or worsen.

Problem

• Fairness interventions
• Early Warning Systems
• Algorithmic fairness
• Procurement constraints
• Fairness theatre

Innovation

• post-hoc fairness interventions
• error-type profiling
• early warning systems
• fairness theatre
• demographic disparities