Efficient Machine Translation Corpus Generation: Integrating Human-in-the-Loop Post-Editing with Large Language Models

2025-02-18

AI Summary

To address the high annotation cost and difficulty in quality assessment inherent in machine translation (MT) corpus construction, this paper proposes a human-in-the-loop semi-automated corpus production framework. Methodologically, it integrates enhanced translation synthesis, LLM-driven pseudo-labeling, assistive annotation analysis, and real-time post-editing feedback to establish a dynamically optimized training loop. By jointly modeling large language models (LLMs) and MT quality estimation, the framework enables high-fidelity pseudo-label generation and precise translation recommendation; a lightweight human–machine interface further lowers the barrier to expert intervention. Experimental results demonstrate that, under reduced total cost, post-editing quality improves by 12.7% and manual annotation effort decreases by 38%. The framework is open-sourced to support reproducible and extensible research.

Technology Category

• Humans and AI: Human-in-the-loop Machine Learning
• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Code Generation / Program Synthesis from Natural Language

Application Category

• Economics, Online Markets and Human Computation: Humans versus LLMs for data annotation and labeling
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This paper introduces an advanced methodology for machine translation (MT) corpus generation, integrating semi-automated, human-in-the-loop post-editing with large language models (LLMs) to enhance efficiency and translation quality. Building upon previous work that utilized real-time training of a custom MT quality estimation metric, this system incorporates novel LLM features such as Enhanced Translation Synthesis and Assisted Annotation Analysis, which improve initial translation hypotheses and quality assessments, respectively. Additionally, the system employs LLM-Driven Pseudo Labeling and a Translation Recommendation System to reduce human annotator workload in specific contexts. These improvements not only retain the original benefits of cost reduction and enhanced post-edit quality but also open new avenues for leveraging cutting-edge LLM advancements. The project's source code is available for community use, promoting collaborative developments in the field. The demo video can be accessed here.

Problem

• Enhancing machine translation corpus generation efficiency
• Integrating human-in-the-loop post-editing with LLMs
• Reducing human annotator workload using LLM-driven methods

Innovation

• Human-in-the-loop post-editing
• Large Language Models integration
• LLM-Driven Pseudo Labeling