Measuring the Influence of Incorrect Code on Test Generation

2024-09-14

AI Summary

This study systematically quantifies, for the first time, how the correctness of the target code affects the quality of test generation by large language models (LLMs). Method: We conduct an empirical analysis across 11 mainstream LLMs, 3 widely adopted benchmarks (HumanEval, MBPP, etc.), and 41 real-world open-source repositories. Results: Incorrect code substantially degrades test accuracy (−57%), line coverage (−12%), and defect detection rate (−47%). In contrast, augmenting the input with natural-language code descriptions mitigates these losses, improving accuracy by 18%, coverage by 4%, and defect detection by 34%. Our contributions are threefold: (1) the first quantitative characterization of how code correctness impacts LLM-based test generation; (2) validation that this effect holds consistently across synthetic benchmarks and real-world codebases; and (3) identification of natural-language code descriptions as an effective, practically deployable mitigation strategy—providing both theoretical grounding and actionable guidance for LLM-driven testing practice.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Computer Vision: Large Vision Models

Application Category

• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

It is natural to suppose that a Large Language Model is more likely to generate correct test cases when prompted with correct code under test, compared to incorrect code under test. However, the size of this effect has never been previously measured, despite its obvious importance for both practicing software engineers and researchers. To answer the question, we conducted a comprehensive empirical study on 5 open source and 6 closed source language models, with 3 widely-used benchmark data sets together with 41 repo-level real-world examples from two different real-world data sets. Our results reveal that, when compared to incorrect code under test, LLMs prompted with correct code achieve improvements in test accuracy, code coverage, and bug detection of 57%, 12%, and 24% respectively. We further show that these scientific conclusions carry over from the three benchmark data sets to the real-world code, where tests generated for incorrect code experience a 47% worse bug detection rate. Finally, we report that improvements of +18% in accuracy, +4% coverage, and +34% in bug detection can be achieved by providing natural language code descriptions. These findings have actionable conclusions. For example, the 47% reduction in real-world bug detection is a clear concern. Fortunately, it is a concern for which our findings about the added value of descriptions offer an immediately actionable remedy.

Problem

• Measure impact of incorrect code on LLM test generation accuracy
• Compare test performance between correct and incorrect code inputs
• Evaluate effectiveness of natural language descriptions for improvement

Innovation

• Empirical study on 11 LLMs with benchmarks
• Correct code improves test accuracy significantly
• Natural language descriptions enhance bug detection