🤖 AI Summary
This work addresses the limitations of traditional test case generation methods, which are labor-intensive and poorly scalable, as well as existing large language model (LLM)-based single-prompt approaches that often suffer from hallucination and redundancy, leading to insufficient branch coverage. The authors propose a two-stage automated test generation framework: in the first stage, static analysis extracts input constraints to generate initial test cases; in the second stage, dual agents leveraging line and branch feedback iteratively refine prompts based on uncovered metadata, while a redundancy-prevention caching mechanism avoids duplicate test generation. The approach is language-agnostic, supporting multiple programming languages such as C and Python, and achieves significant improvements in both line and branch coverage on standard benchmarks with linearly scalable execution time.
📝 Abstract
Traditional approaches to test case generation often involve manual effort and incur significant computational overhead. Additionally, these approaches are not scalable, and hence, unsuitable for complex software systems. Recently, Large Language Models (LLMs) have been applied to software testing. However, single-shot prompt engineering-based approaches tend to hallucinate and generate redundant test cases, resulting in fewer branches. To handle the above-mentioned limitations, in this paper, we propose FeedbackLLM, a novel automated language-agnostic test case generation framework based on tightly coupled two-stage approach. In the first stage, FeedbackLLM extracts the input constraints by parsing source code and generates the possible test cases. The quality of the test cases is evaluated in the second stage by the following two specialized LLM feedback agents: (i) Line Feedback Agent: extracts the metadata related to missed line executions and (ii) Branch Feedback Agent: extracts the metadata of the unexecuted branch conditions. The above agents operate in a two-stage process, communicating in tandem, and this procedure is repeated for k-steps. Further, we also introduced a redundancy prevention cache to avoid duplicate API requests and avoid unnecessary execution cycles. The performance of the proposed architecture is evaluated on the standard benchmark programs related to C and Python programs. FeedbackLLM demonstrated more line and branch coverage than baseline tools while scaling linearly in execution time.