Studying quantization trade-offs for efficient inference deployment in machine translation

2026-07-31

AI Summary

This study addresses the lack of systematic evaluation of how quantization impacts inference efficiency and translation quality under orchestrated workloads in real-world server deployments of large language models, particularly given that mainstream benchmarks overlook long-context dynamics. We present the first controlled assessment of EuroLLM and Hy-MT2 (1.7B–22B parameters) on A100/H100 GPUs, combining W4A8/W8A8 quantization with document chunking strategies. Introducing a document-level evaluation based on WMT24++, we uncover quantization–long-context interactions invisible to sentence-level benchmarks. Our experiments reveal that Hy-MT2 maintains robustness under quantization, whereas EuroLLM suffers significant quality degradation. Jointly optimizing chunking and quantization substantially improves the latency–throughput Pareto frontier, demonstrating that the trade-off between translation quality and efficiency is highly dependent on model architecture, quantization format, and chunking strategy.

Abstract

Deploying large language models in realistic server environments poses challenges, as the system needs to provide high-quality responses with low latency. Quantization is a common approach to reduce the memory footprint and improve inference efficiency, yet its impact on latency and throughput is rarely evaluated under controlled, orchestration-level workloads. In this work we study the quantization trade-offs of two translation model families, EuroLLM \citep{martins2025eurollm} and Hy-MT2 \citep{zheng2026hy} across five models ranging from 1.7B to 22B for efficient deployment on a single A100 or H100 GPU. We demonstrate that combining a document-chunking strategy with W4A8 or W8A8 quantization improves the latency-throughput Pareto-curve under a wide range of workloads. Furthermore, since standard machine translation (MT) benchmarks rely on isolated sentences and fail to capture long-context dynamics, we introduce a document-level evaluation from WMT24++ to assess how text chunking strategies affect translation quality under quantization. Our results reveal that standard segment-level evaluation can fail to predict the interaction between quantization and long-context document translation. While Hy-MT2 remains robust under quantization, EuroLLM shows strong sensitivity and translation quality collapses rapidly for all considered quantization formats. Overall, our experiments show that the trade-off between inference efficiency and translation quality depends not only on the quantization format, but also on the choice of text chunking strategy.

Problem

• quantization
• machine translation
• inference efficiency
• long-context
• deployment

Innovation

• quantization
• document-level evaluation
• latency-throughput trade-off
• text chunking
• efficient inference