How Many Bits Can an Adapter Write? Measuring the Capacity and Memorization of Parameter-Efficient Fine-Tuning

2026-07-23

AI Summary

This work challenges the prevailing assumption that parameter-efficient fine-tuning methods such as LoRA encode only “skills” rather than memorized data, by precisely quantifying—in bits—the amount of information written into adapters while the backbone model remains frozen. Leveraging compression-based memory analysis and information-theoretic measures on the Qwen2.5 model, the study reveals that each trainable parameter stores only a few bits of information, with MLP layers exhibiting substantially higher storage efficiency than attention layers. Memory capacity is found to be governed primarily by parameter location rather than sheer parameter count. Furthermore, adapters trained via supervised learning pose notable privacy risks due to data memorization, whereas those trained through reinforcement learning retain almost no secrets from the original training data.

Abstract

A LoRA adapter is a few megabytes that almost everyone treats as a skill rather than a record of the data behind it. We put that assumption on a scale. Extending compression-based memorization analysis to the frozen-base setting, we measure directly, in bits, how much a low-rank adapter writes into a model it never changes. The answer is both smaller than full fine-tuning and less lawful than parameter counting would predict. Adapters store a couple of bits per trainable parameter, well short of a full model's budget, but that figure turns less on how many parameters an adapter carries than on where they sit. Move the same parameter budget from attention into the MLP and it holds nearly twice as much; strip the frozen base of its structure and the capacity all but disappears. Applied to realistic fine-tunes of Qwen2.5, the same instrument shows privacy leakage rising with the bits an adapter writes rather than the parameters it nominally has, and it draws a clean line between supervised and reinforcement learning: the secrets that supervised fine-tuning copies down verbatim, an adapter trained on verifiable rewards never records. Measuring what fine-tuning writes, rather than attacking it after the fact, turns a piece of folklore into a quantity one can design against.

Problem

• parameter-efficient fine-tuning
• memorization
• information capacity
• LoRA adapter
• frozen-base models

Innovation

• parameter-efficient fine-tuning
• memorization measurement
• information capacity
• LoRA adapter
• privacy leakage