RESTOR: Automated Test Oracle Generation for RESTful APIs via Reinforcement Learning

2026-07-26

AI Summary

This work addresses the challenge of generating reliable test oracles for black-box testing of RESTful APIs in agile development environments, where documentation and logs are often scarce. The authors propose a lightweight fine-tuning approach for large language models based on Group Relative Policy Optimization, augmented with a novel data augmentation strategy that enables the model to acquire testing “common sense.” This allows the model to accurately identify semantically stable response fields and generate robust assertions. Evaluated on 246 real-world services, the method achieves an F₁ score of 85.42% and, when deployed in ByteDance’s production environment, increases the adoption rate of automatically generated test cases from 74.1% to over 96%, substantially reducing manual QA effort.

Abstract

Modern REST API testing faces a critical challenge in defining reliable test oracles, particularly in agile industrial environments where formal specifications (e.g., OpenAPI) are frequently missing or outdated, and historical execution logs are unavailable for newly deployed endpoints. In this paper, we present Restor (Reinforcement Enhanced Single-Traffic Oracle generator for REST APIs), a framework that generates executable test assertions from a single observed request-response pair in a black-box setting. Unlike existing approaches that rely on rule-based templates or massive training logs, Restor utilizes a novel data augmentation pipeline to fine-tune a lightweight Large Language Model (LLM) via Group Relative Policy Optimization (GRPO). This training process enables the model to internalize testing "common sense" by optimizing a reward function that jointly encourages: (i) the selection of stable, semantically meaningful fields for validation and the avoidance of dynamic noise (e.g., timestamps or trace IDs); (ii) the generation of robust assertions that withstand logic variations. We evaluate Restor on an industrial dataset comprising over 2,300 API traces across 246 real-world services. Comprehensive experiments demonstrate that Restor significantly outperforms prompt-engineered baselines and generalist models, achieving a superior $F_1$ score of 85.42% in key field identification and increasing the proportion of semantically accurate assertions. Furthermore, deployment in a production CI/CD workflow at ByteDance confirms its practical value: the system raised the adoption rate of automatically generated test cases from 74.1% to over 96%, substantially reducing manual Quality Assurance (QA) effort while ensuring high execution stability.

Problem

• test oracle
• RESTful APIs
• automated testing
• specification gap
• black-box testing

Innovation

• Reinforcement Learning
• Test Oracle Generation
• RESTful APIs
• Large Language Model
• Black-box Testing