Forget to Generalize: Iterative Adaptation for Generalization in Federated Learning

2026-02-04

AI Summary

This work addresses the significant degradation in generalization performance of federated learning under non-independent and identically distributed (Non-IID) data. To mitigate this issue, the authors propose Iterative Federated Adaptation (IFA), a novel training paradigm that incorporates a “generational forgetting and evolution” mechanism. Within this framework, a subset of model parameters is periodically reinitialized—either randomly or by layer—across generational training rounds. This strategy enables the model to escape local optima while preserving globally effective representations, all without altering the underlying federated algorithm. IFA seamlessly integrates into existing federated learning frameworks and demonstrates substantial improvements, achieving an average 21.5% increase in global accuracy across benchmarks including CIFAR-10, MIT-Indoors, and Stanford Dogs, with particularly pronounced gains in highly heterogeneous Non-IID settings.

Technology Category

• Machine Learning: Distributed Machine Learning & Federated Learning
• Computer Vision: Generative Adversarial Networks (GANs) for Vision
• Natural Language Processing: Generation

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The Web is naturally heterogeneous with user devices, geographic regions, browsing patterns, and contexts all leading to highly diverse, unique datasets. Federated Learning (FL) is an important paradigm for the Web because it enables privacy-preserving, collaborative machine learning across diverse user devices, web services and clients without needing to centralize sensitive data. However, its performance degrades severely under non-IID client distributions that is prevalent in real-world web systems. In this work, we propose a new training paradigm - Iterative Federated Adaptation (IFA) - that enhances generalization in heterogeneous federated settings through generation-wise forget and evolve strategy. Specifically, we divide training into multiple generations and, at the end of each, select a fraction of model parameters (a) randomly or (b) from the later layers of the model and reinitialize them. This iterative forget and evolve schedule allows the model to escape local minima and preserve globally relevant representations. Extensive experiments on CIFAR-10, MIT-Indoors, and Stanford Dogs datasets show that the proposed approach improves global accuracy, especially when the data cross clients are Non-IID. This method can be implemented on top any federated algorithm to improve its generalization performance. We observe an average of 21.5%improvement across datasets. This work advances the vision of scalable, privacy-preserving intelligence for real-world heterogeneous and distributed web systems.

Problem

• Federated Learning
• Non-IID
• Generalization
• Heterogeneous Data
• Web Systems

Innovation

• Iterative Federated Adaptation
• Forget and Evolve
• Non-IID Generalization
• Federated Learning
• Parameter Reinitialization