Evaluating Prompt Scope and Demonstration Similarity in Local LLM Machine Translation

2026-07-28

AI Summary

This study addresses the limited scope of current evaluations of large language models (LLMs) in machine translation, which often rely on a single prompt format and lack systematic analysis of multilingual target requests and example selection strategies. The work proposes a novel evaluation framework for local LLMs that treats prompt scope—monolingual versus language-family-level multilingual—and example retrieval strategy—random, lexical similarity, or embedding similarity—as key variables. Using the FLORES dataset under zero-shot and 5-shot settings, the authors evaluate LLaMA3.2-3B, Mistral, Qwen2.5-14B, and dedicated MT systems. Results show that dedicated MT systems consistently outperform LLMs; few-shot prompting improves Mistral and Qwen2.5 but degrades LLaMA3.2 performance; embedding-based retrieval slightly surpasses other strategies; and while language-family-level prompting is viable, smaller models are prone to structured output errors.

Abstract

Large language models (LLMs) are increasingly used as general-purpose translation systems, but their behavior is usually evaluated under a single prompt shape: translate one source sentence into one target language. In practice, users may ask for one target language, for several related languages at once, or for translations conditioned on examples. This paper studies prompt scope and demonstration selection as experimental variables for local LLM machine translation. We evaluate English-to-Romance and English-to-Germanic translation on the full FLORES devtest split for nine official European Union languages. We compare three local instruction-tuned LLMs, llama3.2:3b, mistral:latest, and qwen2.5:14b, against dedicated MT baselines from OPUS-MT and NLLB-200. We test zero-shot prompting and k=5 few-shot prompting with random, lexical-similarity, and embedding-similarity demonstration selection. We also compare single-target prompts with JSON-formatted family-scope prompts that request all languages in a family at once. Results show that dedicated MT systems remain strongest overall, especially for Germanic languages. Few-shot prompting helps mistral:latest and qwen2.5:14b, but hurts llama3.2:3b; embedding retrieval is best on average for the stronger LLMs, but its advantage over random and lexical examples is modest. Family-scope prompting is feasible for stronger local LLMs but exposes structured-output failures in smaller models. These findings motivate evaluating LLM translation not only by language pair and metric, but also by prompt scope, retrieval strategy, and multi-target compliance.

Problem

• prompt scope
• demonstration similarity
• local LLM
• machine translation
• few-shot prompting

Innovation

• prompt scope
• demonstration selection
• local LLM
• few-shot machine translation
• multi-target translation