Do Coverage and Mutation Scores of LLM-Generated Test Suites Correlate with Their Effectiveness? (Replicability Study)

2026-07-24

AI Summary

Existing research often relies on code coverage and mutation score as proxy metrics to evaluate the effectiveness of test cases generated by large language models (LLMs), yet their correlation with actual fault-detection capability remains unclear. This study conducts a large-scale empirical analysis of test suites produced by diverse LLMs across varied testing scenarios, systematically examining the relationships among coverage, mutation score, and real defect detection performance. The findings reveal that the validity of these proxy metrics is highly context-dependent: they offer some predictive value in regression testing but prove unreliable when the target code contains faults. Furthermore, the size of the test suite has limited influence on these correlations. These results challenge conventional assumptions in test evaluation and provide new empirical grounding for assessing LLM-generated tests.

Abstract

Recent advances in large language models (LLMs) have driven growing interest in using LLMs to automate test generation. Prior work commonly evaluates generated test suites using proxy metrics such as code coverage and mutation score. However, studies by Inozemtseva et al. and Papadakis et al. show that, for human-written tests, correlations among coverage, mutation, and real-bug detection can largely vanish once test suite size is controlled, raising concerns about the validity of evaluations based on proxy metrics. It also remains unclear whether these conclusions carry over to LLM-generated tests, given that prevailing LLM-based test-generation workflows differ substantially from traditional approaches. In this paper, we conduct a large-scale replication study of these two prior works using a wide range of test suites generated by a diverse set of LLMs, and re-examine the relationships among coverage, mutation, and real-bug detection effectiveness. Our findings diverge substantially from prior results. We show that the usefulness of coverage and mutation is highly context-dependent: in regression-style settings where the code provided to the LLM can be reasonably assumed bug-free, these metrics can provide meaningful signals when comparing across models; in another common scenario where the code-under-test may already be buggy and the goal is to expose the bug within the code-under-test, they no longer serve as reliable indicators. We also find little evidence that test suite size is a dominant confounder for correlations among coverage, mutation, and real-bug detection for LLM-generated tests. Based on these findings, we discuss how to interpret results from prior studies and provide actionable guidance for evaluating LLM-based test generation.

Problem

• LLM-generated test suites
• code coverage
• mutation score
• real-bug detection
• test effectiveness

Innovation

• LLM-generated test suites
• coverage
• mutation score
• bug detection effectiveness
• replication study