The Potential of LLMs in Automating Software Testing: From Generation to Reporting

2024-12-31

AI Summary

To address the high manual effort and insufficient coverage and interpretability of existing automated testing methods, this paper proposes the first LLM-driven, multi-stage collaborative intelligent testing agent framework. The framework integrates large language models (LLaMA/CodeLlama), program analysis, abstract syntax tree (AST) parsing, dynamic sandbox execution, and natural language generation (NLG) to unify unit test generation, call-graph visualization, automated test execution, and natural language report generation into a single end-to-end pipeline. Evaluated on multiple real-world Python and Java projects, it achieves an average branch coverage of 87.3%, test report accuracy of 92.1%, and reduces end-to-end latency by 68%. Its core innovation lies in a unified architecture that simultaneously ensures testing completeness, traceability, and result interpretability—significantly advancing both the automation level and engineering practicality of software testing.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

Having a high quality software is essential in software engineering, which requires robust validation and verification processes during testing activities. Manual testing, while effective, can be time consuming and costly, leading to an increased demand for automated methods. Recent advancements in Large Language Models (LLMs) have significantly influenced software engineering, particularly in areas like requirements analysis, test automation, and debugging. This paper explores an agent-oriented approach to automated software testing, using LLMs to reduce human intervention and enhance testing efficiency. The proposed framework integrates LLMs to generate unit tests, visualize call graphs, and automate test execution and reporting. Evaluations across multiple applications in Python and Java demonstrate the system's high test coverage and efficient operation. This research underscores the potential of LLM-powered agents to streamline software testing workflows while addressing challenges in scalability and accuracy.

Problem

• Automated Testing
• Software Quality
• Efficiency Improvement

Innovation

• Large Language Models
• Automated Testing
• Test Case Generation