Evaluating the cognitive reality of Spanish irregular morphomic patterns: Humans vs. Transformers

2025-07-29

AI Summary

This study investigates the cognitive plausibility of Spanish irregular morpheme patterns. Methodologically, it adopts, for the first time, an analysis framework aligned with human behavioral experiments to systematically compare Transformer language models against human participants on a word-formation task. Input verb frequency (natural, low-frequency, high-frequency) was experimentally manipulated to assess model sensitivity to phonological similarity and the proportion of irregular verbs in training data. Results show that while models achieve higher stem and affix accuracy than humans, their response preferences significantly diverge from human behavior. Phonological sensitivity aligns with human performance only under natural and low-frequency conditions; moreover, model behavior is strongly modulated by the prevalence of irregular verbs in training corpora. These findings reveal fundamental limitations of current Transformer models in capturing human-like linguistic cognition and provide novel evidence regarding the cognitive interpretability of neural language representations.

Technology Category

• Natural Language Processing: Lexical Semantics and Morphology
• Cognitive Modeling & Cognitive Systems: Simulating Human Behavior
• Humans and AI: Learning Human Values and Preferences

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

This study investigates the cognitive plausibility of the Spanish irregular morphomic pattern by directly comparing transformer-based neural networks to human behavioral data from citet{Nevins2015TheRA}. Using the same analytical framework as the original human study, we evaluate whether transformer models can replicate human-like sensitivity to a complex linguistic phenomena, the morphome, under controlled input conditions. Our experiments focus on three frequency conditions: natural, low-frequency, and high-frequency distributions of verbs exhibiting irregular morphomic patterns. While the models outperformed humans in stem and suffix accuracy, a clear divergence emerged in response preferences. Unlike humans, who consistently favored natural responses across all test items, models' preferred irregular responses and were influenced by the proportion of irregular verbs in their training data. Additionally, models trained on the natural and low-frequency distributions, but not the high-frequency distribution, were sensitive to the phonological similarity between test items and real Spanish L-shaped verbs.

Problem

• Comparing transformer models to human data on Spanish irregular morphomic patterns
• Assessing model sensitivity to linguistic phenomena under varying frequency conditions
• Analyzing divergence in response preferences between humans and AI models

Innovation

• Transformer models vs human data comparison
• Controlled input conditions for linguistic analysis
• Training data frequency impacts model responses