ASTER: Natural and Multi-language Unit Test Generation with LLMs

2024-09-04

AI Summary

To address critical challenges in multilingual (Java/Python) unit test generation—including poor test readability, low coverage, and frequent compilation failures—this paper proposes a static-analysis-driven large language model (LLM) testing paradigm. Our method integrates multilingual abstract syntax tree (AST) parsing, environment mocking, coverage-guided prompt engineering, and static analysis feedback to impose structured constraints on LLM outputs and enable iterative refinement. Evaluated on large-scale industrial Java/Python codebases and standard benchmarks, our approach achieves branch coverage competitive with or surpassing state-of-the-art (SOTA) tools; notably, it is the first to demonstrate effectiveness in Python. A user study (N=161) confirms that generated tests are significantly more natural, readable, and stylistically consistent with human-written tests.

Technology Category

• Natural Language Processing: (Large) Language Models
• Machine Learning: Large Multimodal Models (LMMs)
• Planning, Routing, and Scheduling: Planning with Language Models

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web search

Abstract

Implementing automated unit tests is an important but time-consuming activity in software development. To assist developers in this task, many techniques for automating unit test generation have been developed. However, despite this effort, usable tools exist for very few programming languages. Moreover, studies have found that automatically generated tests suffer poor readability and do not resemble developer-written tests. In this work, we present a rigorous investigation of how large language models (LLMs) can help bridge the gap. We describe a generic pipeline that incorporates static analysis to guide LLMs in generating compilable and high-coverage test cases. We illustrate how the pipeline can be applied to different programming languages, specifically Java and Python, and to complex software requiring environment mocking. We conducted an empirical study to assess the quality of the generated tests in terms of code coverage and test naturalness -- evaluating them on standard as well as enterprise Java applications and a large Python benchmark. Our results demonstrate that LLM-based test generation, when guided by static analysis, can be competitive with, and even outperform, state-of-the-art test-generation techniques in coverage achieved while also producing considerably more natural test cases that developers find easy to understand. We also present the results of a user study, conducted with 161 professional developers, that highlights the naturalness characteristics of the tests generated by our approach.

Problem

• Automated Unit Testing
• Multi-language Environment
• Low Coverage

Innovation

• ASTER
• Large Language Models
• Static Analysis for Code Coverage