Revisiting the Role of Natural Language Code Comments in Code Translation

2026-01-23

AI Summary

Existing code translation benchmarks commonly overlook natural language comments and lack systematic evaluation of their impact. This work addresses this gap by conducting over 80,000 controlled experiments on more than 1,100 code samples spanning five programming languages—C, C++, Go, Java, and Python—and demonstrates, for the first time at scale, that comments—particularly those describing overall functionality—play a crucial role in improving translation accuracy. Building on this insight, we propose COMMENTRA, a method that effectively enhances the code translation performance of large language models. Experimental results show that COMMENTRA nearly doubles the approximate accuracy of translated code, underscoring the importance of leveraging natural language context in code translation tasks.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Language and Vision

Application Category

• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work

Abstract

The advent of large language models (LLMs) has ushered in a new era in automated code translation across programming languages. Since most code-specific LLMs are pretrained on well-commented code from large repositories like GitHub, it is reasonable to hypothesize that natural language code comments could aid in improving translation quality. Despite their potential relevance, comments are largely absent from existing code translation benchmarks, rendering their impact on translation quality inadequately characterised. In this paper, we present a large-scale empirical study evaluating the impact of comments on translation performance. Our analysis involves more than $80,000$ translations, with and without comments, of $1100+$ code samples from two distinct benchmarks covering pairwise translations between five different programming languages: C, C++, Go, Java, and Python. Our results provide strong evidence that code comments, particularly those that describe the overall purpose of the code rather than line-by-line functionality, significantly enhance translation accuracy. Based on these findings, we propose COMMENTRA, a code translation approach, and demonstrate that it can potentially double the performance of LLM-based code translation. To the best of our knowledge, our study is the first in terms of its comprehensiveness, scale, and language coverage on how to improve code translation accuracy using code comments.

Problem

• code translation
• natural language comments
• translation accuracy
• programming languages
• code benchmarks

Innovation

• code translation
• natural language comments
• large language models
• COMMENTRA
• empirical study