🤖 AI Summary
This work addresses a critical vulnerability in existing experience replay mechanisms for continual learning: their susceptibility to stealthy attacks that manipulate replay sample selection without altering the data itself. We demonstrate for the first time that an adversary with access only to replay indices can severely degrade model performance by skewing the class distribution of replayed samples. To this end, we propose an audit-aware stealthy attack framework that balances invisibility and impact. Our method estimates per-class utility via lightweight metrics—either EMA loss or confidence—and optimizes the replay distribution using projection techniques based on KL divergence (via exponential tilting) or total variation distance (via mass redistribution). A sliding-window scheduler further adapts the attack to rolling audits. Experiments across multiple continual learning benchmarks show significant drops in final accuracy and backward transfer, with the KL-based variant achieving high destructiveness while remaining highly evasive under diverse auditing mechanisms.
📝 Abstract
Continual learning (CL) models often use experience replay to reduce catastrophic forgetting, but their robustness to replay sampling interference remains underexplored. Existing CL attacks alter inputs or training pipelines (poisoning/backdoors) and rarely include explicit auditable constraints, limiting realism. Here, auditability means a monitor can verify compliance from sampler-visible telemetry - e.g., logged replay index/label statistics - by checking that the realized replay class histogram stays close to a nominal baseline and that replay rate is unchanged per batch and/or over a rolling window. We study a limited-privilege insider who controls only replay index selection, not pixels, labels, or model parameters, while staying within auditable limits such as queue priorities. We introduce Amnesia, a replay composition attack that maximizes degradation under two budgets: a visibility budget delta bounding the TV/KL divergence from a nominal class histogram p0, and a mass budget f fixing the replay rate. Amnesia has two steps: (i) compute lightweight class utilities, such as EMA loss or confidence, to tilt p0 toward harmful classes; and (ii) project the tilt back into the delta-ball using efficient KL (exponential tilt) or TV (balanced mass redistribution) optimizers. A windowed scheduler enforces rolling audits. Across challenging CL benchmarks and strong replay baselines, Amnesia consistently lowers final accuracy (ACC) and worsens backward transfer (-BWT). The KL variant delivers high impact while remaining largely undetected under multiple audit schemes, including per-batch and rolling-window checks. The TV variant is more damaging but easier to detect, especially under tight per-class constraints. These results expose index-only replay control as a practical, auditable threat surface in CL systems and establish a principled impact-visibility trade-off.