Canonicalized Stable-List Replay for Private Federated Continual Learning over Language-Model Embeddings

2026-05-29

AI Summary

This work addresses the challenges of federated continual learning under user-level differential privacy, where clients face unordered, noisy replay buffers that are difficult to align across devices. To overcome these issues, the authors propose CSLR, a method wherein clients privately generate candidate replay distributions in a shared sentence embedding space. The server leverages signatures derived from public anchor sentences to align these distributions across clients, thereby resolving identifiability issues in unordered replay lists without requiring additional replay data. The approach provides theoretical guarantees relating the number of anchors to identification success rates. Experimental results demonstrate that, under a privacy budget of ε=4, CSLR achieves average performance gains of 3.9–5.6 points over the strongest baseline across multiple NLP continual learning benchmarks, significantly outperforming Hungarian matching and optimal transport-based methods.

Abstract

Federated continual learning (FCL) lets distributed clients adapt language-model heads to evolving NLP tasks without sharing raw text. Under user-level differential privacy (DP), replay-based continual learning faces a structural obstacle: clients can release only small noisy lists of candidate replay summaries, and those lists are unordered across clients. We introduce Canonicalized Stable-List Replay (CSLR), where clients privately produce candidate replay distributions over a shared sentence-embedding space and the server aligns them using signatures induced by public anchor sentences. The anchors provide identifiability for aggregation rather than additional replay data. We prove that, under an observable anchor-signature margin, $O(\log(N/η)/p)$ anchors distinguish $N$ candidate list elements with probability at least $1-η$, and we give a scoped anchorless non-identifiability result for unordered-label oracle models. Across five seeds on continual classification, NER, and dialogue benchmarks, CSLR improves the final average task metric by 3.9--5.6 points over the strongest non-CSLR DP baseline at $\eps=4$ under the reported replay-release budget, while also outperforming Hungarian and optimal-transport matchers. The formal privacy guarantee covers replay release; end-to-end private training additionally requires composition with a private optimizer for task-head updates.

Problem

• Federated Continual Learning
• Differential Privacy
• Replay-based Learning
• Unordered Lists
• Language-Model Embeddings

Innovation

• Canonicalized Stable-List Replay
• Federated Continual Learning
• Differential Privacy
• Sentence Embedding Alignment
• Anchor-based Aggregation