APITestGenie: Generating Web API Tests from Requirements and API Specifications with LLMs

2026-04-02

AI Summary

This work addresses the challenges of manual Web API integration testing, which is time-consuming, error-prone, and often misaligned with business requirements. The authors propose a novel approach that synergistically combines large language models (LLMs), retrieval-augmented generation (RAG), and prompt engineering to jointly parse natural language business requirements and OpenAPI specifications, thereby automatically generating executable test scripts that are both semantically meaningful and syntactically correct. Evaluated on ten real-world APIs, the method successfully produced valid tests for 89% of the business requirements within three attempts, uncovered multiple previously unknown integration defects, and substantially reduced the manual effort required for test development.

Abstract

Modern software systems rely heavily on Web APIs, yet creating meaningful and executable test scripts remains a largely manual, time-consuming, and error-prone task. In this paper, we present APITestGenie, a novel tool that leverages Large Language Models (LLMs), Retrieval-Augmented Generation (RAG), and prompt engineering to automatically generate API integration tests directly from business requirements and OpenAPI specifications. We evaluated APITestGenie on 10 real-world APIs, including 8 APIs comprising circa 1,000 live endpoints from an industrial partner in the automotive domain. The tool was able to generate syntactically and semantically valid test scripts for 89\% of the business requirements under test after at most three attempts. Notably, some generated tests revealed previously unknown defects in the APIs, including integration issues between endpoints. Statistical analysis identified API complexity and level of detail in business requirements as primary factors influencing success rates, with the level of detail in API documentation also affecting outcomes. Feedback from industry practitioners confirmed strong interest in adoption, substantially reducing the manual effort in writing acceptance tests, and improving the alignment between tests and business requirements.

Problem

• Web API testing
• test generation
• manual testing
• API integration tests
• business requirements

Innovation

• Large Language Models
• Retrieval-Augmented Generation
• API Test Generation
• OpenAPI Specification
• Prompt Engineering