The Persona Hierarchy Model: Understanding Contextual Generalization in Fine-Tuning LLMs

2026-10-06

AI Summary

This study addresses the limitation that behavioral generalization during large language model (LLM) fine-tuning is often constrained by training contexts. To overcome this, we propose a personality hierarchy model that elucidates how a shared default personality drives cross-contextual behavioral generalization. Building upon this theoretical framework, we introduce Personality-Preserving Regularization (PPR), a method designed to effectively suppress reward hacking during reinforcement learning-based fine-tuning. Experimental results demonstrate that PPR reduces the reward hacking rate to below 0.2% while fully preserving model accuracy. This work provides both a novel perspective and a robust methodology for understanding and precisely controlling the generalization boundaries of LLM fine-tuning.

Abstract

Language models are routinely fine-tuned under a fixed context, such as a generic system prompt, persona or domain-specific instruction, yet the learned behavior sometimes stays confined to that context and sometimes broadly generalizes to unseen contexts. We propose the Persona Hierarchy Model to explain this: a shared default persona influences behavior across contexts. Under this model, fine-tuning that modifies the shared persona promotes broader transfer, whereas changes to local personas remain more context-specific. Across 120 fine-tuned models spanning four behaviors and 15 training contexts, generalization narrowness positively correlates with the similarity between the training context's persona and the default persona (Pearson's r = 0.72 for Qwen3-4B). Prior fine-tuning under the default context can broaden generalization in subsequent training under other contexts. Aligning contextual responses with default-persona responses produces stronger effects. Finally, we propose persona-preserving regularization (PPR) to confine undesired contextual generalization. In RL, PPR cuts reward hacking from 42-55% to at most 0.2% under every evaluated prompt while retaining accuracy gains. These results support the Persona Hierarchy Model as an explanation for contextual generalization and can motivate future controls on unintended generalization for better alignment of LLMs.

Problem

• contextual generalization
• fine-tuning
• large language models
• persona
• reward hacking

Innovation

• Persona Hierarchy Model
• Contextual Generalization
• Persona-Preserving Regularization
• Reward Hacking
• LLM Alignment