Combining TSL and LLM to Automate REST API Testing: A Comparative Study

2025-09-05

AI Summary

REST API testing faces challenges including high distributed-system complexity, diverse usage scenarios, substantial manual effort for test design, and insufficient coverage. This paper proposes an automated test case generation method integrating a Test Specification Language (TSL) with large language models (LLMs), leveraging customized prompt engineering and an end-to-end automation pipeline to efficiently transform OpenAPI specifications into executable test cases. Its key contribution is the novel use of TSL as a structured intermediate representation that bridges LLM-based reasoning and formal test semantics, thereby enhancing generation accuracy and verifiability. Experimental evaluation demonstrates that Claude 3.5 Sonnet significantly outperforms other mainstream LLMs in test generation success rate, API path coverage, and mutation score—achieving an average 37% improvement in coverage and reducing manual authoring effort by approximately 62%.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Planning, Routing, and Scheduling: Planning with Language Models
• Machine Learning: Large Multimodal Models (LMMs)

Application Category

• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

The effective execution of tests for REST APIs remains a considerable challenge for development teams, driven by the inherent complexity of distributed systems, the multitude of possible scenarios, and the limited time available for test design. Exhaustive testing of all input combinations is impractical, often resulting in undetected failures, high manual effort, and limited test coverage. To address these issues, we introduce RestTSLLM, an approach that uses Test Specification Language (TSL) in conjunction with Large Language Models (LLMs) to automate the generation of test cases for REST APIs. The approach targets two core challenges: the creation of test scenarios and the definition of appropriate input data. The proposed solution integrates prompt engineering techniques with an automated pipeline to evaluate various LLMs on their ability to generate tests from OpenAPI specifications. The evaluation focused on metrics such as success rate, test coverage, and mutation score, enabling a systematic comparison of model performance. The results indicate that the best-performing LLMs - Claude 3.5 Sonnet (Anthropic), Deepseek R1 (Deepseek), Qwen 2.5 32b (Alibaba), and Sabia 3 (Maritaca) - consistently produced robust and contextually coherent REST API tests. Among them, Claude 3.5 Sonnet outperformed all other models across every metric, emerging in this study as the most suitable model for this task. These findings highlight the potential of LLMs to automate the generation of tests based on API specifications.

Problem

• Automating REST API test case generation
• Addressing test scenario creation challenges
• Overcoming exhaustive input combination testing limitations

Innovation

• Combining TSL and LLM for REST API test automation
• Using prompt engineering with automated evaluation pipeline
• Generating tests from OpenAPI specifications via LLMs