Localized Adaptation Reveals Distinct Learning Signatures in Transformers

2026-07-28

AI Summary

This work addresses the challenge that during Transformer fine-tuning, learning signals are diffusely distributed across the entire network, making it difficult to precisely control what the model learns and how it generalizes. To tackle this, the authors propose a hierarchical LoRA-based localized fine-tuning strategy, which applies adaptation modules separately to early, middle, and late layers. They systematically investigate the acquisition, transferability, and boundary characteristics of five distinct learning objectives. For the first time, they define and empirically validate that different tasks exhibit unique “adaptation geometry” patterns, demonstrating that adaptation location is a critical design variable for steering model learning behavior. Controlled experiments across five major model families reveal consistent and reproducible layer-wise adaptation preferences across tasks, offering both theoretical grounding and practical guidance for efficient, targeted fine-tuning.

Abstract

Transformer adaptation is typically distributed across model depth, even when the intended change is narrow. We investigate how adaptation site shapes what a model learns, how well that learning generalizes, and how selectively it is applied. We introduce a controlled benchmark spanning five objectives (lexical binding, factual association, behavioral policy learning, causal mapping, and procedural reasoning) and define each objective's "adaptation geometry" as its profile of acquisition, transfer, and boundedness under full-stack and early-, middle-, or late-layer LoRA. The objectives exhibit distinct geometries. Lexical binding favors early-layer adaptation for acquisition and boundedness but requires broader updates for transfer; factual association favors later layers among localized adapters; behavioral learning separates late-layer action acquisition from middle-layer policy gating; and causal and procedural transfer benefit most from middle- or full-stack adaptation. These patterns largely persist under parameter-matched controls, and most corresponding directional contrasts replicate across five model families. These findings establish adaptation site as a key design variable for controlling what models learn, generalize, and leave unchanged.

Problem

• localized adaptation
• learning signatures
• adaptation geometry
• transfer learning
• parameter efficiency

Innovation

• adaptation geometry
• localized adaptation
• LoRA
• transfer learning
• layer-specific learning