Probing Character-level Transformers for the Spanish L-shaped Morphome

2026-08-04

AI Summary

This study investigates how character-level Transformers encode morphological knowledge of Spanish L-verbs—irregular verbs exhibiting stem alternations. Through probing analyses, lemma-separated cross-validation, and comparisons against surface-form baselines, the internal representations of 60 models across five architectures are systematically evaluated. The findings reveal that L-verb alternations are significantly decodable from model representations and consistently outperform all surface-form baselines, even in the absence of observed alternation instances during training. This indicates that models encode L-verb forms as lemma-specific abstract representations rather than merely memorizing surface strings. These abstract representations are predominantly localized in the decoder layers at positions corresponding to stem-final consonants. This work provides the first evidence that character-level Transformers possess lexical abstraction capabilities extending beyond surface forms, although their generalization mechanisms remain less robust than those observed in human language acquisition.

Abstract

When a transformer learns an irregular morphological pattern, what has it learned? Our test case is the Spanish \emph{L-shaped morphome}, a complex irregular pattern in which the verb's stem alternates in exactly the first-person singular indicative and all subjunctive forms, and whose membership no phonological, semantic, or syntactic feature predicts. Prior studies have shown that character-level transformers can reproduce this pattern, but that evidence describes what models produce, not what they represent. Probing five architectures, twelve trained models each, under lemma-disjoint cross-validation with controls and surface baselines, we show that the models encode the L-shaped class itself, not just its visible alternations. It is decodable above every surface baseline, survives instances in which every form shows the same stem, and probes trained on alternating instances still classify non-alternating ones. The encoding is localized where the stem choice is made, at the stem-final consonant position of the middle decoder, before the alternant is read. And it is item-specific: which verbs a model learned matters far more than which architecture it is. The models store the morphome as an item-specific lexical abstraction, sufficient to reproduce the pattern but not to generalize it as humans do.

Problem

• morphome
• irregular morphology
• character-level transformers
• lexical abstraction
• Spanish verb conjugation

Innovation

• morphome
• character-level transformers
• probing
• lexical abstraction
• L-shaped alternation