Modern Models, Medieval Texts: A POS Tagging Study of Old Occitan

2025-03-10

AI Summary

This study systematically evaluates open-source large language models (LLMs) on part-of-speech (POS) tagging for Old Occitan—a low-resource, medieval language characterized by nonstandard orthography and diachronic grammatical variation. We conduct the first zero-shot and few-shot POS tagging experiments on authentic Old Occitan texts from religious hagiography and medical treatises, benchmarking multiple state-of-the-art open-source LLMs. Results reveal substantial performance degradation compared to high-resource languages, with errors predominantly concentrated in morphological inflection and orthographic variants. To address this, we propose a fine-grained error taxonomy and actionable optimization strategies. Our work fills a critical gap in empirical LLM evaluation for historical linguistics, establishing the first rigorous benchmark for NLP on highly variable, low-resource medieval languages and offering methodological insights for robust modeling under orthographic and grammatical instability.

Technology Category

• Natural Language Processing: Other
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Evaluation and Analysis

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics

Abstract

Large language models (LLMs) have demonstrated remarkable capabilities in natural language processing, yet their effectiveness in handling historical languages remains largely unexplored. This study examines the performance of open-source LLMs in part-of-speech (POS) tagging for Old Occitan, a historical language characterized by non-standardized orthography and significant diachronic variation. Through comparative analysis of two distinct corpora-hagiographical and medical texts-we evaluate how current models handle the inherent challenges of processing a low-resource historical language. Our findings demonstrate critical limitations in LLM performance when confronted with extreme orthographic and syntactic variability. We provide detailed error analysis and specific recommendations for improving model performance in historical language processing. This research advances our understanding of LLM capabilities in challenging linguistic contexts while offering practical insights for both computational linguistics and historical language studies.

Problem

• Evaluates LLMs for POS tagging in Old Occitan.
• Assesses handling of orthographic and syntactic variability.
• Provides recommendations for historical language processing.

Innovation

• Evaluates LLMs for Old Occitan POS tagging
• Compares hagiographical and medical text corpora
• Provides error analysis and improvement recommendations