Evaluating Large Language Models for Diacritic Restoration in Romanian Texts: A Comparative Study

2025-11-17

AI Summary

Accented character restoration in diacritic-rich languages like Romanian poses challenges for NLP, particularly under low-resource and zero-/few-shot settings. Method: We systematically evaluate leading LLMs—GPT-4o, Gemini, Llama-3, and Mixtral—on diacritic restoration in Romanian, using a large-scale native corpus and multiple prompt templates under zero- and few-shot configurations. Contribution/Results: This is the first quantitative analysis of how model architecture, pretraining data scale, and prompting strategy affect accent recovery accuracy. GPT-4o significantly outperforms all baselines (12.6% higher accuracy, lower variance), whereas open-weight models exhibit higher performance variability. Optimized prompting yields an average 9.3% accuracy gain. Our work establishes a reproducible methodology and empirical benchmark for lightweight NLP adaptation to tone- or diacritic-sensitive low-resource languages.

Technology Category

Machine Learning: Large Multimodal Models (LMMs)Natural Language Processing: Language Grounding & Multi-modal NLPPlanning, Routing, and Scheduling: Planning with Language Models

Application Category

User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendationSearch and Retrieval-Augmented AI: Large language models for searchEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Automatic diacritic restoration is crucial for text processing in languages with rich diacritical marks, such as Romanian. This study evaluates the performance of several large language models (LLMs) in restoring diacritics in Romanian texts. Using a comprehensive corpus, we tested models including OpenAI's GPT-3.5, GPT-4, GPT-4o, Google's Gemini 1.0 Pro, Meta's Llama 2 and Llama 3, MistralAI's Mixtral 8x7B Instruct, airoboros 70B, and OpenLLM-Ro's RoLlama 2 7B, under multiple prompt templates ranging from zero-shot to complex multi-shot instructions. Results show that models such as GPT-4o achieve high diacritic restoration accuracy, consistently surpassing a neutral echo baseline, while others, including Meta's Llama family, exhibit wider variability. These findings highlight the impact of model architecture, training data, and prompt design on diacritic restoration performance and outline promising directions for improving NLP tools for diacritic-rich languages.

Problem

• Evaluating LLMs for diacritic restoration in Romanian texts
• Comparing performance across multiple models and prompt templates
• Analyzing impact of architecture and training on restoration accuracy

Innovation

• Evaluated multiple large language models for diacritic restoration
• Used prompt templates from zero-shot to multi-shot instructions
• Analyzed impact of model architecture and training data