Fair Finetuning Mitigates Distribution Inference Attacks

2026-06-01

AI Summary

This work addresses the risk that machine learning models trained on sensitive data may leak group-level distributional information, enabling adversaries to infer subgroup proportions via black-box queries—a threat known as distribution inference attacks. To mitigate this, the authors propose Fair Fine-tuning (FFt), a method that fine-tunes pretrained models on complementary distribution samples while enforcing Equalized Odds fairness constraints. The study establishes, for the first time, a tight theoretical bound linking the magnitude of Equalized Odds disparity to the adversary’s attack advantage, demonstrating that fairness directly suppresses distributional leakage and thereby forging a novel pathway for joint fairness-and-privacy defenses. Experiments across six multimodal datasets (tabular, image, and text) show that FFt substantially reduces attack effectiveness, consistently lowering the accuracy gap to below 0.1—for instance, from 15% to 4% on the ACS Income dataset.

Abstract

Machine learning models trained on sensitive data can inadvertently leak population-level information about their training distributions -- a threat known as distribution inference attack (DIA). An adversary with black-box access can infer sensitive demographic properties, such as subgroup proportions, without observing any training data directly. While defenses such as differential privacy and property unlearning have been proposed, the link between fairness constraints and distributional leakage remains unexplored. We propose Fair Fine-tuning (FFt): a trained model is fine-tuned on samples from the complementary distribution under an Equalized Odds (EO) constraint. We provide a complete theoretical characterization, proving the tight bound $\text{Adv}(\mathcal{A},M_f) \le Δ_{\text{EO}} \cdot W$, where $W$ quantifies how distinguishable the two training distributions are by their sensitive-attribute composition. We also establish a necessary condition for FFt to reduce adversarial advantage and prove tightness of the bound. We evaluate across six datasets spanning tabular (ACS Income, COMPAS, German Credit), image (UTKFaces), and NLP (Bias in Bios) modalities. Rehearsal-based FFt consistently reduces the adversarial accuracy gap below the detection threshold $τ!=!0.1$ across all settings; on ACS Income, the gap falls from $\sim!15%$ to under $4%$. Our work provides the first formal bound connecting a model's measured EO disparity directly to its adversarial advantage in the DIA game, opening a new avenue for unified fairness-and-privacy defenses.

Problem

• distribution inference attack
• fairness
• privacy leakage
• sensitive attributes
• adversarial advantage

Innovation

• Fair Fine-tuning
• Distribution Inference Attack
• Equalized Odds
• Adversarial Advantage
• Privacy-Fairness Tradeoff