Regional Tiny Stories: Using Small Models to Compare Language Learning and Tokenizer Performance

2025-04-07

AI Summary

This work addresses the evaluation and optimization of small language models (SLMs) for low-resource Indian regional languages—Hindi, Marathi, and Bengali—where data scarcity and linguistic complexity hinder effective modeling. Method: We introduce the first trilingual TinyStories dataset for these languages, combining human translation with LLM-synthesized narratives. We extend the TinyStories framework to Indian languages via language-specific tokenizers, information-theoretic and morphological complexity analysis, and cross-lingual joint modeling of syntactic accuracy, narrative coherence, and creativity. Contribution/Results: Experiments show that parameter-efficient SLMs (~10M parameters) can effectively model regional Indian languages; language-specific tokenization yields significant performance gains; LLM-synthesized data outperforms human-translated data; and Hindi models achieve the highest overall performance. The study establishes a reproducible data curation paradigm, tokenizer design methodology, and multilingual evaluation framework for training SLMs on low-resource languages.

Technology Category

Natural Language Processing: (Large) Language ModelsMachine Learning: Large Multimodal Models (LMMs)Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSearch and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchWeb Mining and Content Analysis: Large pretrained models with web data

Abstract

Small Language Models (SLMs) offer efficient alternatives to LLMs for specific domains. The 2023 TinyStories study developed an English dataset that allows SLMs with 1 to 10 million parameters to produce coherent outputs. Our research expands this framework by translating the original dataset into Indian languages and creating synthetic data using LLMs. We focus on Hindi, Marathi, and Bengali, evaluating SLMs for regional language processing and understanding linguistic complexity. We show that SLMs efficiently process regional languages with significantly fewer parameters than LLMs, providing a complementary framework for ``inference based evaluation"of tokenization strategies and linguistic complexity. Our analysis shows that language-specific tokenizers outperform general-purpose ones for Indian languages. Empirical validations, supported by information-theoretic and morphological analyses, provides fundamental understanding behind the better performance of Hindi models over Marathi and Bengali. Additionally, we show that synthetic datasets outperform translated content for training SLMs. Correlation analyses reveal cross-linguistic patterns and language-specific relationships between creativity, grammatical precision, and narrative completeness. These findings advance both the practical application of SLMs to underserved languages and our theoretical understanding of neural language development.

Problem

• Evaluate SLMs for regional language processing in Hindi, Marathi, Bengali.
• Compare language-specific vs general-purpose tokenizers for Indian languages.
• Assess synthetic vs translated datasets for training SLMs.

Innovation

• Expanded TinyStories dataset to Indian languages
• Used synthetic data from LLMs for SLM training
• Language-specific tokenizers outperform general ones