Private Model Personalization Revisited

2025-06-23

AI Summary

This work addresses model personalization under user-level differential privacy (DP) in shared-representation frameworks, focusing on privately recovering shared embeddings and local low-dimensional representations for statistically heterogeneous users in federated learning, while controlling excess risk. Methodologically, we propose a private federated algorithm built upon FedRep, integrating Johnson–Lindenstrauss transforms to compress both embeddings and user data dimensions. Theoretical contributions include: (1) the first input-dimension-independent margin-based risk bound for binary classification under user-level DP; (2) an improved privacy error term—reduced from $widetilde{O}(d^2k)$ to $widetilde{O}(dk)$—applicable to broader sub-Gaussian user distributions; and (3) robust generalization guarantees even under label noise. Empirical evaluation demonstrates substantial improvements in accuracy and in the privacy–utility trade-off compared to baselines.

Technology Category

• Machine Learning: Privacy
• Natural Language Processing: Safety and Robustness
• Search and Optimization: Learning to Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Security and Privacy: Security and privacy of machine learning and AI applications
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

We study model personalization under user-level differential privacy (DP) in the shared representation framework. In this problem, there are $n$ users whose data is statistically heterogeneous, and their optimal parameters share an unknown embedding $U^* inmathbb{R}^{d imes k}$ that maps the user parameters in $mathbb{R}^d$ to low-dimensional representations in $mathbb{R}^k$, where $kll d$. Our goal is to privately recover the shared embedding and the local low-dimensional representations with small excess risk in the federated setting. We propose a private, efficient federated learning algorithm to learn the shared embedding based on the FedRep algorithm in [CHM+21]. Unlike [CHM+21], our algorithm satisfies differential privacy, and our results hold for the case of noisy labels. In contrast to prior work on private model personalization [JRS+21], our utility guarantees hold under a larger class of users' distributions (sub-Gaussian instead of Gaussian distributions). Additionally, in natural parameter regimes, we improve the privacy error term in [JRS+21] by a factor of $widetilde{O}(dk)$. Next, we consider the binary classification setting. We present an information-theoretic construction to privately learn the shared embedding and derive a margin-based accuracy guarantee that is independent of $d$. Our method utilizes the Johnson-Lindenstrauss transform to reduce the effective dimensions of the shared embedding and the users' data. This result shows that dimension-independent risk bounds are possible in this setting under a margin loss.

Problem

• Private recovery of shared embedding in federated learning
• Handling heterogeneous user data under differential privacy
• Dimension-independent risk bounds for binary classification

Innovation

• Private federated learning for shared embeddings
• Johnson-Lindenstrauss transform reduces dimensions
• Differential privacy under sub-Gaussian distributions