A Framework for Double-Blind Federated Adaptation of Foundation Models

2025-02-03

AI Summary

To address the challenge of securely adapting large models to downstream tasks under conditions of data decentralization, privacy sensitivity, and constrained edge-device resources, this paper proposes a Dual-Blind Federated Adaptation (DBFA) framework. The framework achieves mutual invisibility between data holders and model providers for the first time: it ensures communication privacy via fully homomorphic encryption and secure aggregation; constructs a homomorphically computable proxy model through knowledge distillation; introduces a backpropagation-free, low-rank parallel adapter to minimize on-device computational and memory overhead; and incorporates a privacy-preserving permutation mechanism to resist model extraction attacks. Extensive experiments on four image classification benchmarks demonstrate that DBFA maintains high accuracy—average accuracy degradation <1.2%—while providing strong privacy guarantees against white-box adversaries and enabling deployment on memory-constrained edge devices.

Technology Category

• Machine Learning: Privacy
• Computer Vision: Adversarial Attacks & Robustness
• Natural Language Processing: Safety and Robustness

Application Category

• Security and Privacy: Privacy-enhancing technologies
• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking

Abstract

The availability of foundational models (FMs) pre-trained on large-scale data has advanced the state-of-the-art in many computer vision tasks. While FMs have demonstrated good zero-shot performance on many image classification tasks, there is often scope for performance improvement by adapting the FM to the downstream task. However, the data that is required for this adaptation typically exists in silos across multiple entities (data owners) and cannot be collated at a central location due to regulations and privacy concerns. At the same time, a learning service provider (LSP) who owns the FM cannot share the model with the data owners due to proprietary reasons. In some cases, the data owners may not even have the resources to store such large FMs. Hence, there is a need for algorithms to adapt the FM in a double-blind federated manner, i.e., the data owners do not know the FM or each other's data, and the LSP does not see the data for the downstream tasks. In this work, we propose a framework for double-blind federated adaptation of FMs using fully homomorphic encryption (FHE). The proposed framework first decomposes the FM into a sequence of FHE-friendly blocks through knowledge distillation. The resulting FHE-friendly model is adapted for the downstream task via low-rank parallel adapters that can be learned without backpropagation through the FM. Since the proposed framework requires the LSP to share intermediate representations with the data owners, we design a privacy-preserving permutation scheme to prevent the data owners from learning the FM through model extraction attacks. Finally, a secure aggregation protocol is employed for federated learning of the low-rank parallel adapters. Empirical results on four datasets demonstrate the practical feasibility of the proposed framework.

Problem

• Distributed Learning
• Privacy Protection
• Intellectual Property Rights

Innovation

• Differential Privacy
• Secure Multi-party Computation
• Federated Learning