TQLite: Multi-LLM Jury Guided Distillation for Real-time MQM Translation Quality Evaluation

2026-08-03

AI Summary

This work addresses the limitations of small language models (SLMs) in translation quality estimation, where they struggle with complex reasoning, and the prohibitive computational costs of large reasoning models (LRMs), despite their superior performance. To bridge this gap, the authors propose TQLite, a distillation framework that leverages a multi-LRM jury to generate high-quality synthetic data. This data is then used to fine-tune SLMs through response aggregation and knowledge distillation strategies. Experimental results demonstrate that SLMs trained with TQLite achieve significantly better performance on MQM-based evaluation metrics compared to off-the-shelf SLMs, closely approaching the accuracy of state-of-the-art LRMs while retaining the efficiency and low computational cost inherent to smaller models.

Abstract

Large language models (LLMs) have demonstrated impressive performance in MQM-based translation quality (TQ) evaluation, and recent advances in large reasoning models (LRMs) promise even greater improvements. However, both LLMs and LRMs are computationally expensive to deploy at scale, while small language models (SLMs)---though much more efficient---struggle with the complex reasoning required for evaluation tasks. In this work, we present an extensive empirical study benchmarking SLMs, LLMs, and LRMs across a wide range of TQ evaluation setups, providing a comprehensive view of the current landscape and establishing best practices. To address the scalability challenge, we introduce TQLite, a novel distillation framework that enables SLMs to approach the MQM evaluation performance of the best LRM-based evaluators. Our approach leverages a multi-LRM jury to generate high-quality synthetic training data via practical data curation techniques and aggregation of evaluation responses across a diverse panel of models. Our results demonstrate that SLMs trained via TQLite achieve strong MQM evaluation performance that far exceeds off-the-shelf evaluation capabilities of standard SLMs, offering a scalable and cost-effective alternative to LLM- and LRM-based evaluators.

Problem

• translation quality evaluation
• MQM
• model distillation
• scalability
• small language models

Innovation

• multi-LLM jury
• distillation framework
• MQM evaluation
• small language models
• synthetic training data