π€ AI Summary
Existing testing approaches struggle to effectively detect functional bugs that do not cause program crashes: manually written unit tests are costly, heuristic-based test generation lacks semantic understanding, and fuzzing relies heavily on crash signals. To address this limitation, this work proposes LISA, a novel framework that uniquely integrates large language models (LLMs) with program invariants. LISA employs semantically guided API call sequence generation and an API n-gramβbased feedback mechanism to iteratively refine test cases. The approach substantially improves both the detection rate and precision of functional defects, outperforming state-of-the-art fuzzing techniques and LLM-driven testing methods in terms of code coverage and the generation of high-confidence bug reports.
π Abstract
Manually writing unit tests to uncover functional bugs in software libraries is not only time-consuming but also requires a deep understanding of the intended semantics of the APIs. Heuristic-based test generation methods suffer from low usability because they cannot reason about program semantics or interpret source code and documentation as humans do. Traditional fuzzing techniques like OSS-Fuzz often rely on crashes to detect bugs, but functional bugs do not always cause crashes. To overcome these limitations, we present LISA, a novel LLM-based invariant testing framework for software functional bugs. LISA iteratively generates API sequences and program invariants guided by API n-gram feedback, achieving higher bug-detection rates and competitive code coverage compared with both fuzzing and prior LLM-based test generation approaches, and reporting each finding as a high-confidence bug candidate for developer confirmation.