Less is More: Pre-Training Cross-Lingual Small-Scale Language Models with Cognitively-Plausible Curriculum Learning Strategies

2026-04-11

AI Summary

This study addresses the insufficient cognitive plausibility and weak cross-lingual generalization of small-scale language models (SSLMs) in cross-lingual pretraining. Methodologically, we propose a curriculum learning framework grounded in child language acquisition theory: (1) constructing an age-stratified, language-specific, child-directed phonetic corpus; (2) computationally formalizing empirically grounded acquisition theories—spanning four typologically diverse language families (e.g., Growing, Inwards, MMM)—into fine-grained, learnable curriculum objectives for the first time; and (3) integrating curriculum learning with cross-lingual modeling. Our key contribution is the unification of cognitive interpretability and modeling performance. Experiments on the BabyLM benchmark demonstrate that our approach significantly outperforms non-curricular baselines, and that fine-grained, acquisition-driven curriculum strategies consistently enhance the cross-lingual generalization capability of SSLMs.

Technology Category

Natural Language Processing: (Large) Language ModelsMachine Learning: Large Multimodal Models (LMMs)Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSearch and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchSemantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs

Abstract

Curriculum Learning has been a popular strategy to improve the cognitive plausibility of Small-Scale Language Models (SSLMs) in the BabyLM Challenge. However, it has not led to considerable improvements over non-curriculum models. We assess whether theoretical linguistic acquisition theories can be used to specify more fine-grained curriculum learning strategies, creating age-ordered corpora of Child-Directed Speech for four typologically distant language families to implement SSLMs and acquisition-inspired curricula cross-lingually. Comparing the success of three objective curricula (Growing, Inwards and MMM) that precisely replicate the predictions of acquisition theories on a standard SSLM architecture, we find fine-grained acquisition-inspired curricula can outperform non-curriculum baselines and performance benefits of curricula strategies in SSLMs can be derived by specifying fine-grained language-specific curricula that precisely replicate language acquisition theories.

Problem

• Improving Small-Scale Language Models' cognitive plausibility.
• Applying linguistic theories to refine curriculum learning strategies.
• Cross-lingual implementation of acquisition-inspired curricula in SSLMs.

Innovation

• Uses fine-grained curriculum learning strategies
• Applies age-ordered corpora across languages
• Replicates language acquisition theories precisely