Robust Knowledge Distillation in Federated Learning: Counteracting Backdoor Attacks

2025-02-01

AI Summary

Existing robust defenses against backdoor attacks launched by malicious clients in federated learning often rely on strong assumptions—such as uniform data distribution and known fractions of adversaries—rendering them ineffective under realistic conditions characterized by statistical heterogeneity and dynamically varying numbers of attackers. Method: This paper proposes the first lightweight, three-tier defense framework integrating clustering-based anomaly detection, trustworthy model selection, and knowledge distillation—operating without any prior assumptions on data distribution or adversary count. It adaptively identifies malicious clients, selects high-quality local models, and distills robust knowledge into the global model. Contribution/Results: Evaluated across multiple heterogeneous datasets and variable attack proportions, our method reduces the backdoor attack success rate (ASR) to <3% while incurring less than 1% degradation in global model accuracy—significantly outperforming state-of-the-art defenses.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Multiagent Systems: Adversarial Agents
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• User Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systems
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

Federated Learning (FL) enables collaborative model training across multiple devices while preserving data privacy. However, it remains susceptible to backdoor attacks, where malicious participants can compromise the global model. Existing defence methods are limited by strict assumptions on data heterogeneity (Non-Independent and Identically Distributed data) and the proportion of malicious clients, reducing their practicality and effectiveness. To overcome these limitations, we propose Robust Knowledge Distillation (RKD), a novel defence mechanism that enhances model integrity without relying on restrictive assumptions. RKD integrates clustering and model selection techniques to identify and filter out malicious updates, forming a reliable ensemble of models. It then employs knowledge distillation to transfer the collective insights from this ensemble to a global model. Extensive evaluations demonstrate that RKD effectively mitigates backdoor threats while maintaining high model performance, outperforming current state-of-the-art defence methods across various scenarios.

Problem

• Federated Learning
• Robustness
• Byzantine Attacks

Innovation

• Robust Knowledge Distillation
• Secure Federated Learning
• Malicious Attack Defense