bias auditing and mitigation

The core challenge in data quality monitoring lies in error provenance—specifically, identifying the underlying mechanisms that generate errors—a problem largely overlooked by existing work, which seldom models such mechanisms explicitly. This paper focuses on errors arising from intrinsic dependencies within data and proposes MechDetect, the first method to systematically extend missing-data mechanism detection to diverse error types—including outliers, inconsistencies, and format violations. Leveraging joint statistical modeling and supervised learning, MechDetect simultaneously models tabular data and their error masks to automatically determine whether observed errors stem from inherent characteristics of the original data. Extensive experiments across multiple benchmark datasets demonstrate that MechDetect significantly outperforms state-of-the-art baselines in accurately diagnosing error-generation mechanisms. By providing mechanistic interpretability, it establishes a theoretical foundation and practical framework for explainable data repair.

Current large language model (LLM) protocols rely solely on end-to-end accuracy for evaluation, which fails to reveal when errors are corrected or introduced during reasoning—particularly under distribution shifts or in modular compositions where behavior becomes opaque. This work proposes a dual-outcome measurement interface that records correctness states before and after each protocol step, enabling fine-grained auditing through reusable metrics: correction rate (c) and corruption rate (γ). By integrating paired outcome measurements, difficulty-proxied conditioning, and Markov factorization tests, the method identifies three failure mechanisms—mixed shifts, presentation contamination, and state insufficiency—and supports compositional prediction and dynamic control in multi-step protocols. Experiments on synthetic mathematical tasks and GSM8K demonstrate that the interface accurately predicts optimal activation or suppression of protocol modules, substantially enhancing system predictability and robustness.

Post-processing debiasing methods may inadvertently introduce new forms of unfairness—particularly through overcorrection caused by imbalanced prediction flips across demographic groups. To address this, we propose “Flip Disparity,” a novel metric suite that quantifies, for the first time in post-processing, the relative proportion of predictions flipped per group, thereby overcoming limitations of conventional fairness metrics that ignore transparency and proportionality in correction behavior. Our method leverages differences in confusion matrices and inter-group comparative analysis, integrated within a unified framework combining visual diagnostic tools and strategy-comparability assessment. This paradigm significantly enhances the interpretability of debiasing strategies and enables reliable detection of latent imbalanced corrections. Empirical evaluation across multiple benchmark datasets reveals previously undetected correction biases in widely adopted fairness algorithms. The proposed framework establishes a verifiable, auditable standard for responsible algorithmic governance.

This paper addresses the lack of directionality, statistical rigor, and computational efficiency in Bayesian model misspecification diagnostics. We propose a novel diagnostic framework based on Uniform Parameterization Checks (UPCs). Our core innovation is the first systematic exploitation of the theoretical property that posterior samples should follow the prior distribution under correct model specification; leveraging probability integral transforms, UPCs uniformly map all random components—across prior, likelihood, and data—into independent *u*-values. This enables interpretable, aggregable, statistically rigorous, and computationally efficient diagnostics (≈ cost of one posterior sample) for individual model components (prior, likelihood, data subsets). UPCs support targeted detection of specific misspecification types—including dependence structure violations, tail discrepancies, and missing correlations—and accurately localize sources of misspecification in both synthetic and real-world examples. Theoretical analysis establishes consistency of the proposed tests.

To address the challenge of sustaining fairness auditing for deployed models under data policy drift, subgroup heterogeneity, and dynamic distributional shifts, this paper proposes the first dynamic fairness auditing framework tailored for streaming data. Methodologically, it innovatively adapts “testing by betting”—a game-theoretic statistical paradigm—into fairness monitoring, integrating anytime-valid inference with nonparametric, sample-size-agnostic sequential testing that adapts to time-varying sampling strategies. Unlike conventional batch-based approaches, our framework enables real-time, continuous fairness assessment without requiring prior knowledge of data distribution or fixed sample sizes. Experiments on three benchmark datasets demonstrate that the method achieves high detection sensitivity, low false positive rate, and strong robustness against concept drift and subgroup imbalance. These properties collectively enhance the practicality, timeliness, and deployability of fairness monitoring in real-world operational environments.

This study addresses the limited efficacy of existing bias mitigation techniques in predictive models trained on government data, which often fail to achieve fairness objectives. Using crime rate prediction by Bristol City Council as a case study, the work systematically investigates the root causes of this failure, attributing it to inherent structural and historical biases embedded in the data rather than flaws in model design. Through cross-cutting fairness experiments employing multiple mainstream predictive models and fairness interventions, the analysis examines dimensions such as data distribution shifts, accumulation of historical bias, and reporting delays. The findings reveal the inadequacy of approaches that address only single sensitive attributes and demonstrate that current technical methods are insufficient to eliminate the deeply entrenched inequities in governmental datasets, thereby offering critical empirical evidence and a cautionary insight for algorithmic deployment in public policy contexts.

This study addresses the pervasive issue of data errors in real-world databases—such as missing values, redundancy, statistical biases, and outliers—which significantly degrade downstream analytical and machine learning performance. Recognizing that existing taxonomies are incomplete and terminology inconsistent, this work presents the first unified framework that integrates traditional data errors with statistically oriented inaccuracies critical in the AI era. It proposes a non-overlapping tripartite classification structure—comprising missing, erroneous, and redundant data—and systematically constructs a comprehensive catalog of 35 distinct error types. Through formal definitions, illustrative examples, and a thorough literature review, the paper establishes standardized terminology and precise characterizations, thereby offering a clear, rigorous theoretical foundation and practical toolkit for data quality assessment and cleaning.

This work addresses the challenge of systematic performance disparities exhibited by machine learning models across sensitive subgroups in high-stakes settings, where existing fairness auditing methods often rely on strong distributional assumptions or incur prohibitive computational costs. The paper introduces, for the first time, empirical likelihood into fairness auditing, proposing a nonparametric, distribution-free statistical inference framework. By constructing a discrepancy statistic that asymptotically follows a mixture of chi-squared distributions, the method enables efficient and accurate fairness certification and subgroup bias detection without resampling. Demonstrating both computational efficiency and statistical power, the approach successfully identifies intersectional biases on the COMPAS dataset, achieves coverage matching nominal confidence levels, and accelerates computation by several orders of magnitude compared to bootstrap methods, making it suitable for large-scale deployment.

This study investigates the compatibility conflict between statistical parity and equalized odds fairness criteria under scenarios characterized by base rate imbalance across sensitive groups or unreliable labeling. Through probabilistic modeling and theoretical analysis, it systematically demonstrates— for the first time—how disparities in base rates inherently preclude the simultaneous satisfaction of both fairness notions, and derives necessary and sufficient conditions for their incompatibility. The work further advocates for assessing base rate imbalance risks prior to adopting statistical parity, thereby offering a rigorous theoretical foundation and practical guidance for the design of fair algorithms, policy formulation, and compliance auditing.

Current approaches to evaluating bias in large language models often suffer from an imbalance between ecological validity and statistical rigor—either relying on synthetic prompts lacking real-world representativeness or failing to scale effectively. This work proposes a scalable, named-entity-based bias auditing framework that leverages entity probes to measure structural disparities in model outputs at scale and uses synthetic data to replicate bias patterns observed in real-world texts. The framework enables the first billion-scale, multidimensional bias audit, analyzing 1.9 billion data points. It reveals systematic model preferences for left-leaning politicians, Western nations, and corporations, alongside discrimination against the defense and pharmaceutical sectors. Furthermore, while instruction tuning mitigates bias, model scaling exacerbates it, and non-English prompts fail to eliminate the models’ Western-centric tendencies.