Optimal differentially private kernel learning with random projection

2025-07-23

AI Summary

This paper investigates the minimax optimality of differentially private kernel learning within the empirical risk minimization (ERM) framework. For square loss and Lipschitz-smooth convex losses, we propose a novel differentially private kernel learning algorithm based on random projection and Gaussian process modeling, integrating objective perturbation with local strong convexity analysis—bypassing reliance on noisy gradient mechanisms. We establish, for the first time, dimension-free generalization upper bounds, overcoming performance bottlenecks inherent in existing dimensionality-reduction approaches. Theoretically, the algorithm achieves minimax-optimal excess risk. Empirically, it significantly enhances statistical utility and generalization performance under constrained privacy budgets, thereby improving the privacy–utility trade-off.

Technology Category

• Machine Learning: Privacy
• Search and Optimization: Learning to Search
• Natural Language Processing: Learning & Optimization for NLP

Application Category

• Security and Privacy: Security and privacy of machine learning and AI applications
• User Modeling, Personalization and Recommendation: User privacy protection in personalized systems
• Graph Algorithms and Modeling for the Web: Algorithms and analysis for incomplete, noisy, or partially observed Web-related graphs

Abstract

Differential privacy has become a cornerstone in the development of privacy-preserving learning algorithms. This work addresses optimizing differentially private kernel learning within the empirical risk minimization (ERM) framework. We propose a novel differentially private kernel ERM algorithm based on random projection in the reproducing kernel Hilbert space using Gaussian processes. Our method achieves minimax-optimal excess risk for both the squared loss and Lipschitz-smooth convex loss functions under a local strong convexity condition. We further show that existing approaches based on alternative dimension reduction techniques, such as random Fourier feature mappings or $ell_2$ regularization, yield suboptimal generalization performance. Our key theoretical contribution also includes the derivation of dimension-free generalization bounds for objective perturbation-based private linear ERM -- marking the first such result that does not rely on noisy gradient-based mechanisms. Additionally, we obtain sharper generalization bounds for existing differentially private kernel ERM algorithms. Empirical evaluations support our theoretical claims, demonstrating that random projection enables statistically efficient and optimally private kernel learning. These findings provide new insights into the design of differentially private algorithms and highlight the central role of dimension reduction in balancing privacy and utility.

Problem

• Optimizing differentially private kernel learning in ERM framework
• Achieving minimax-optimal excess risk for loss functions
• Improving generalization bounds for private kernel ERM algorithms

Innovation

• Random projection in kernel Hilbert space
• Minimax-optimal excess risk achievement
• Dimension-free generalization bounds derivation