Optimizing Case-Based Reasoning System for Functional Test Script Generation with Large Language Models

2025-03-26

AI Summary

To address repetitive outputs and difficulties in adapting to dynamic code structures when large language models (LLMs) generate functional test scripts, this paper proposes Re4, an enhanced case-based reasoning (CBR) framework built upon a 4R cycle—Retrieve, Reuse, Revise, and Retain—to construct a historical repository of test intents and corresponding scripts. Key contributions include: (1) semantic- and script-structure-aware re-ranking with fine-tuned retrieval; (2) reuse strategy optimization via joint supervised fine-tuning (SFT) and Proximal Policy Optimization (PPO)-based reinforcement learning; and (3) pseudo-labeling to enhance generalization under low-resource conditions. Evaluated on two Huawei Datacom product units, the CBR+Re4 approach achieves significant improvements in script accuracy (+28.6%) and executability (+34.1%), while effectively mitigating redundant generation. The framework demonstrates robust practical efficacy in industrial deployment.

Technology Category

• Planning, Routing, and Scheduling: Replanning and Plan Repair
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Reasoning under Uncertainty: Relational Probabilistic Models

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking

Abstract

In this work, we explore the potential of large language models (LLMs) for generating functional test scripts, which necessitates understanding the dynamically evolving code structure of the target software. To achieve this, we propose a case-based reasoning (CBR) system utilizing a 4R cycle (i.e., retrieve, reuse, revise, and retain), which maintains and leverages a case bank of test intent descriptions and corresponding test scripts to facilitate LLMs for test script generation. To improve user experience further, we introduce Re4, an optimization method for the CBR system, comprising reranking-based retrieval finetuning and reinforced reuse finetuning. Specifically, we first identify positive examples with high semantic and script similarity, providing reliable pseudo-labels for finetuning the retriever model without costly labeling. Then, we apply supervised finetuning, followed by a reinforcement learning finetuning stage, to align LLMs with our production scenarios, ensuring the faithful reuse of retrieved cases. Extensive experimental results on two product development units from Huawei Datacom demonstrate the superiority of the proposed CBR+Re4. Notably, we also show that the proposed Re4 method can help alleviate the repetitive generation issues with LLMs.

Problem

• Optimizing test script generation using LLMs and CBR
• Enhancing CBR system with Re4 for better retrieval and reuse
• Reducing repetitive generation issues in LLM-based test scripts

Innovation

• Utilizes case-based reasoning with 4R cycle
• Introduces Re4 optimization for CBR system
• Employs reinforcement learning for case reuse