English Please: Evaluating Machine Translation for Multilingual Bug Reports

2025-02-20

AI Summary

This study addresses the challenge of translating multilingual bug reports in global software development. We conduct the first systematic evaluation of DeepL, AWS Translate, and ChatGPT on real-world multilingual bug reports from the VS Code “english-please” extension. To this end, we propose the first cross-system machine translation (MT) evaluation framework tailored for software engineering defect reports, integrating five complementary metrics—BLEU, BERTScore, COMET, METEOR, and ROUGE—and publicly release a curated multilingual bug report dataset (gitbugs/multilingual). Results show that DeepL significantly outperforms the others across most metrics, demonstrating superior lexical and semantic alignment for technical text; AWS Translate achieves the highest METEOR score; and ChatGPT lags overall. The study empirically confirms that domain adaptation is critical for accurate technical text translation, providing both empirical evidence and methodological foundations for automated bug triaging.

Technology Category

• Natural Language Processing: Machine Translation, Multilinguality, Cross-Lingual NLP
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning
• Multiagent Systems: Agreement, Argumentation & Negotiation

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Mining multimedia, multimodal, multilingual, cross-lingual Web data
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Accurate translation of bug reports is critical for efficient collaboration in global software development. In this study, we conduct the first comprehensive evaluation of machine translation (MT) performance on bug reports, analyzing the capabilities of DeepL, AWS Translate, and ChatGPT using data from the Visual Studio Code GitHub repository, specifically focusing on reports labeled with the english-please tag. To thoroughly assess the accuracy and effectiveness of each system, we employ multiple machine translation metrics, including BLEU, BERTScore, COMET, METEOR, and ROUGE. Our findings indicate that DeepL consistently outperforms the other systems across most automatic metrics, demonstrating strong lexical and semantic alignment. AWS Translate performs competitively, particularly in METEOR, while ChatGPT lags in key metrics. This study underscores the importance of domain adaptation for translating technical texts and offers guidance for integrating automated translation into bug-triaging workflows. Moreover, our results establish a foundation for future research to refine machine translation solutions for specialized engineering contexts. The code and dataset for this paper are available at GitHub: https://github.com/av9ash/gitbugs/tree/main/multilingual.

Problem

• Evaluates machine translation for bug reports
• Compares DeepL, AWS Translate, ChatGPT performance
• Highlights importance of domain-specific translation accuracy

Innovation

• Evaluates machine translation efficiency
• Compares DeepL, AWS Translate, ChatGPT
• Uses BLEU, BERTScore, COMET metrics