Problem
Research questions and friction points this paper is trying to address.
k-Nearest Neighbor
Machine Translation
Computational Efficiency
Towards Faster k-Nearest-Neighbor Machine Translation
🤖 AI Summary
kNN-MT incurs substantial computational redundancy and high decoding latency due to performing k-nearest-neighbor (kNN) retrieval for every generated token. To address this, we propose Dynamic Skipping Mechanism (DSM), the first learnable, token-level retrieval gating module—implemented as a lightweight MLP—that predicts whether kNN lookup is necessary for each token, thereby executing retrieval only where beneficial. DSM is fully plug-and-play, requiring no architectural modifications to existing kNN-MT systems. Evaluated on standard benchmarks including WMT, DSM reduces kNN retrieval cost by 53% with negligible BLEU degradation (<0.5), while identifying and skipping 67–84% of redundant tokens. This yields significant inference speedup without compromising translation quality. Crucially, DSM is system-agnostic and compatible with all mainstream kNN-MT variants.
📝 Abstract
Recent works have proven the effectiveness of k-nearest-neighbor machine translation(a.k.a kNN-MT) approaches to produce remarkable improvement in cross-domain translations. However, these models suffer from heavy retrieve overhead on the entire datastore when decoding each token. We observe that during the decoding phase, about 67% to 84% of tokens are unvaried after searching over the corpus datastore, which means most of the tokens cause futile retrievals and introduce unnecessary computational costs by initiating k-nearest-neighbor searches. We consider this phenomenon is explainable in linguistics and propose a simple yet effective multi-layer perceptron (MLP) network to predict whether a token should be translated jointly by the neural machine translation model and probabilities produced by the kNN or just by the neural model. The results show that our method succeeds in reducing redundant retrieval operations and significantly reduces the overhead of kNN retrievals by up to 53% at the expense of a slight decline in translation quality. Moreover, our method could work together with all existing kNN-MT systems. This work has been accepted for publication in the jornal Advances in Artificial Intelligence and Machine Learning (ISSN: 2582-9793). The final published version can be found at DOI: https://dx.doi.org/10.54364/AAIML.2024.41111
Innovation
Methods, ideas, or system contributions that make the work stand out.
Multilayer Perceptron Network
k-Nearest Neighbor Machine Translation
Efficiency Improvement
🔎 Similar Papers
No similar papers found.
