Data Collaboration Analysis with Orthonormal Basis Selection and Alignment

2024-03-05

AI Summary

In multi-source machine learning, data collaboration (DC) suffers from sensitivity to target basis selection and a disconnect between theoretical guarantees and empirical performance. To address this, we propose Orthogonal Data Collaboration (ODC), the first framework to incorporate orthogonality constraints into both basis alignment and model training—without exposing raw data or basis representations. This design renders performance nearly invariant to basis selection and transforms alignment into an orthogonal Procrustes problem, admitting a closed-form solution that enables non-iterative, low-overhead federated collaboration. Experiments across multiple benchmarks demonstrate that ODC significantly improves both accuracy and computational efficiency. Moreover, it maintains robust optimality under non-ideal conditions—including statistical heterogeneity—while introducing no additional privacy leakage or communication overhead.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Search and Optimization: Distributed Search

Application Category

• Security and Privacy: Data transparency and provenance
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Data Collaboration (DC) analysis offers a privacy-preserving approach to multi-source machine learning by enabling participants to train a shared model without revealing their raw data. Instead, each participant shares only linearly transformed data through a non-iterative communication protocol, thereby mitigating both privacy risks and communication overhead. The core idea of DC is that while each participant obfuscates their data with a secret linear transformation (or basis), the aggregator aligns these secret bases to a chosen target basis extit{without knowing the secret bases}. Although DC theory suggests that any target basis spanning the same subspace as the secret bases should suffice, empirical evidence reveals that the choice of target basis can substantially influence model performance. To address this discrepancy, we propose extbf{Orthonormal DC (ODC)}, a novel framework that enforces orthonormal constraints during the basis selection and alignment phases. Unlike conventional DC -- which allows arbitrary target bases -- ODC restricts the target to orthonormal bases, rendering the specific choice of basis negligible concerning model performance. Furthermore, the alignment step in ODC reduces to the extbf{Orthogonal Procrustes Problem}, which admits a closed-form solution with favorable computational properties. Empirical evaluations demonstrate that ODC achieves higher accuracy and improved efficiency compared to existing DC methods, aligning with our theoretical findings. Additional evaluations assess performance in non-ideal scenarios with heterogenous distributions, also showing the best overall performance for our method. These findings position ODC as a direct and effective enhancement to current DC frameworks without compromising privacy or communication overhead when orthonormality constraints are applicable.

Problem

• Privacy-preserving multi-source machine learning
• Orthonormal basis selection and alignment
• Enhanced model performance and efficiency

Innovation

• Orthonormal basis selection
• Orthogonal Procrustes Problem
• Privacy-preserving data sharing