ReSCENE: Server-Side Replay for Structural Mitigation of Catastrophic Forgetting in Federated Continual Learning

2026-09-29

AI Summary

This study addresses the challenges of client-side resource constraints and catastrophic forgetting in federated continual learning by proposing the ReSCENE framework. Its core innovation lies in shifting the anti-forgetting mechanism from clients to the server. Specifically, clients upload only lightweight data proxies to reduce communication overhead, while the server employs a temporal aggregation algorithm to compress, store, and jointly train these proxies, thereby optimizing memory efficiency and structurally mitigating forgetting. Experimental results demonstrate that this approach achieves accuracy improvements of up to 31.1 percentage points over baselines across multiple datasets, reduces computational overhead to 0.11×, and decreases communication upload volume to 1/179 of the original amount.

Abstract

Federated continual learning must integrate new tasks over time without losing earlier-task knowledge. Most existing methods attach an anti-forgetting mechanism to the client-trained, server-aggregated loop of federated learning, which holds back new learning to preserve earlier knowledge and burdens resource-constrained clients. We propose ReSCENE, which structurally mitigates catastrophic forgetting by having each client upload a small condensed surrogate of its local data while the server keeps the surrogates of past tasks and trains the global model on them together with the current task surrogates. For efficient server memory, we introduce temporal herding, which selects the more recent surrogates from the pool accumulated over a task into a compressed buffer. Our study provides a theoretical analysis showing that this buffer can represent the original task data more closely than full accumulation of all surrogates. Across CIFAR-10, CIFAR-100, and TinyImageNet, ReSCENE achieves the strongest accuracy over seven baselines, by up to $31.1$ points of average accuracy, while requiring as little as $0.11\times$ of the client computation and up to $179\times$ less upload than the model-update baselines. ReSCENE further demonstrates its effectiveness when scaled to larger client populations and larger models while remaining efficient, which makes it a practical method for federated continual learning.

Problem

• Federated Continual Learning
• Catastrophic Forgetting
• Client Resource Constraints
• Knowledge Retention

Innovation

• Federated Continual Learning
• Catastrophic Forgetting
• Dataset Condensation
• Temporal Herding
• Server-Side Replay