Categorical Prior Lock-in: Why In-Context Learning Fails for Structured Data

2026-06-10

AI Summary

This work investigates the limitations of large language models (LLMs) in generating structured data under distribution shift via in-context learning (ICL). It identifies a previously unreported phenomenon—“category prior locking”—where LLMs fail to adjust the category priors established during pretraining, leading to complete failure on rare categories. The study systematically compares ICL with LoRA fine-tuning on two open-source 7B models: while ICL improves numerical fidelity, it cannot alleviate prior locking; in contrast, LoRA fine-tuning overcomes this limitation but introduces memory leakage and generation instability. These findings reveal a fundamental trade-off between adaptation efficacy through fine-tuning and privacy-preserving generation, highlighting structural constraints in ICL’s ability to adapt to shifted category distributions.

Abstract

Large language models (LLMs) are increasingly used as conditional generators for structured data, relying on in-context learning (ICL) to adapt to new distributions without parameter updates. We investigate the limits of ICL for structured generation under distribution mismatch, using high-cardinality tabular data as a controlled test case, and identify a structural failure mode we term \textit{categorical prior lock-in}: the inability of ICL to update the model's prior over token distributions inherited from pre-training. Across two 7B-parameter open-weight models, ICL improves numerical fidelity with additional examples but exhibits a sharp ceiling on categorical distributions, failing to reproduce rare classes entirely. Parameter-efficient fine-tuning (LoRA) overcomes these limitations but introduces measurable memorization risk and, in some cases, destabilizes structured output generation, highlighting a fundamental trade-off between adaptability and privacy.

Problem

• in-context learning
• structured data
• categorical prior lock-in
• distribution mismatch
• large language models

Innovation

• categorical prior lock-in
• in-context learning
• structured data generation
• distribution mismatch
• parameter-efficient fine-tuning