🤖 AI Summary
This study investigates whether quantization can effectively mitigate the privacy risks posed by large language models’ verbatim reproduction of training data. Departing from conventional membership inference attacks, this work introduces verbatim extraction rate as a novel metric to systematically assess privacy leakage and evaluates the trade-off between memory retention and model performance under various quantization configurations. Using three sizes of the Pythia model family, two quantization algorithms, five precision levels (including 4-bit), and two evaluation corpora, the authors concurrently measure perplexity and verbatim extraction rate. Results show that while quantization reduces memorization more rapidly than it degrades performance, 4-bit quantization in the largest model still retains substantial amounts of training data, highlighting the limitations of quantization as a privacy-preserving technique and revealing its selective forgetting behavior toward memorized content.
📝 Abstract
Language models are almost always quantized before they are deployed, and a growing line of work asks whether quantization also lowers their privacy risk. That work measures privacy almost entirely with membership inference. We think this is the wrong thing to measure for the risk that most people actually worry about, namely a model reproducing its training data word for word, and we measure that directly. Using the Pythia models and the public set of sequences each of them is known to have memorized, we track verbatim extraction across five precision levels, from full precision down to four bits, and across three model sizes, while measuring general capability (perplexity) at every point. We find two things. Quantization is a selective forgetter: verbatim memorization falls off faster than capability at every precision and every model size we tried, and this holds under two unrelated quantization algorithms and two evaluation corpora. But the selectivity is not enough to make quantization a privacy defense, which cuts against the optimistic reading of earlier membership-inference results. At the largest model we study, four-bit quantization still reproduces most of the memorized sequences while giving up only a few percent of capability, and the fraction of memorized data that survives quantization grows with model size. We conclude that compression should not be treated as a way to remove memorized training data, and that extraction, not membership inference, is the number practitioners should be watching. All code, sampled evaluation data, and per-configuration results are released.