Detecting Functional Memorization in Code Language Models

2026-06-10

AI Summary

Existing methods for detecting memorization in large code models rely primarily on textual overlap and struggle to identify functional equivalence when syntactic forms differ. This work proposes a controlled experimental framework that prompts both a model exposed to target code and a baseline model without such exposure using identical function signatures. By integrating execution-based consistency checks with an LLM-as-a-judge mechanism, the approach holistically evaluates both textual and functional similarity of generated outputs. Applying this method to the Olmo-3-32B model yields compelling evidence of functional memorization beyond mere verbatim replication, thereby demonstrating for the first time that large code models can retain and reproduce functionally equivalent code even without surface-level similarity. These findings underscore the urgent need for novel memorization auditing metrics grounded in functional equivalence rather than lexical matching.

Abstract

Large language models (LLMs) are increasingly used to generate code at scale. Meanwhile, prior work has investigated whether training data may be recoverable from model outputs, by auditing the textual overlap between training examples and model generations. Code, however, can be functionally equivalent while textually dissimilar. In this work, we study functional memorization: extraction of functional logic beyond what verbatim metrics detect. We construct a counterfactual setup for Olmo-3-32B, comparing a midtrained model (exposed to target code) against a pretrained reference (not exposed). We prompt both models with Python function signatures and measure both textual and functional similarity (i.e., LLM-as-a-judge, execution-based). Our results show clear evidence of functional memorization, highlighting the need for auditing metrics that go beyond textual overlap.

Problem

• functional memorization
• code language models
• training data extraction
• functional equivalence
• model auditing

Innovation

• functional memorization
• code language models
• counterfactual evaluation
• execution-based similarity
• training data extraction