Out-of-Distribution Federated Distillation with Domain-Aware Proxy

2026-08-09

AI Summary

This work addresses the limited out-of-distribution (OOD) generalization capability of federated distillation by proposing a domain-aware proxy data selection framework. It introduces, for the first time, a domain-aware mechanism into the proxy data optimization process within federated distillation. By synergizing federated learning with knowledge distillation, the method adaptively selects or generates proxy data aligned with the target domain, thereby significantly enhancing model robustness under distribution shifts. Experimental results demonstrate that the proposed approach achieves average accuracies of 82.9% and 80.6% on standard benchmarks, substantially outperforming existing methods. This framework offers a unified and effective solution for OOD generalization in both proxy-data-free and proxy-data-based settings.

Abstract

Federated Learning is a distributed machine learning paradigm that trains a global model by aggregating local clients without sharing private data of each client. Federated Distillation (FD) builds upon this paradigm by leveraging knowledge distillation to exchange soft predictions on proxy data instead of model parameters, enabling more efficient communication and supporting heterogeneous model collaboration. However, FD models trained on In-Distribution data are hardly adapted to Out-of-Distribution (OOD) scenarios. In this paper, we propose a domain-aware proxy selection framework to better adopt proxy data for OOD problems. The experimental results show that the proposed models effectively address the challenges of distribution shifts under OOD with and without proxy data by achieving average 82.9\% and 80.6\% over existing works on standard benchmarks. The codes and data are released in https://anonymous.4open.science/r/DPS-FD-8596.

Problem

• Out-of-Distribution
• Federated Distillation
• Distribution Shift
• Proxy Data
• Federated Learning

Innovation

• Federated Distillation
• Out-of-Distribution
• Domain-Aware Proxy
• Federated Learning
• Knowledge Distillation