Private Prediction via Shrinkage

2026-02-05

AI Summary

This work addresses the challenge of high sample complexity in differentially private online prediction, where traditional methods scale as $\sqrt{T}$ with the number of queries $T$, limiting scalability. The authors propose a novel approach based on a shrinking mechanism that, for the first time, reduces the sample complexity to polylogarithmic in $T$ while preserving $(\varepsilon,\delta)$-differential privacy. By integrating techniques from online learning, VC dimension theory, and differential privacy, the resulting private predictor achieves a sample complexity of $\widetilde{O}(\mathrm{VC}(\mathcal{C})^{3.5} \log^{3.5} T)$ for any concept class $\mathcal{C}$ under the forgetful setting. For halfspaces in $\mathbb{R}^d$ against adaptive adversaries, the bound improves to $\widetilde{O}(d^{5.5} \log T)$, significantly outperforming existing results.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Learning to Search
• Reasoning under Uncertainty: Stochastic Optimization

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: Fairness, privacy, and diversity in economic environments
• Security and Privacy: Privacy-enhancing technologies

Abstract

We study differentially private prediction introduced by Dwork and Feldman (COLT 2018): an algorithm receives one labeled sample set $S$ and then answers a stream of unlabeled queries while the output transcript remains $(\varepsilon,\delta)$-differentially private with respect to $S$. Standard composition yields a $\sqrt{T}$ dependence for $T$ queries. We show that this dependence can be reduced to polylogarithmic in $T$ in streaming settings. For an oblivious online adversary and any concept class $\mathcal{C}$, we give a private predictor that answers $T$ queries with $|S|= \tilde{O}(VC(\mathcal{C})^{3.5}\log^{3.5}T)$ labeled examples. For an adaptive online adversary and halfspaces over $\mathbb{R}^d$, we obtain $|S|=\tilde{O}\left(d^{5.5}\log T\right)$.

Problem

• differentially private prediction
• online queries
• sample complexity
• privacy composition
• streaming setting

Innovation

• differentially private prediction
• sample complexity
• online learning
• adaptive adversary
• VC dimension