Flaky Test Recognition when Testing CPSs Using Hybrid Models

2026-08-06

AI Summary

This work addresses the challenge of “flaky tests” in cyber-physical systems (CPS), which arise from environmental uncertainties and timing synchronization issues, leading to unreliable test outcomes. Building upon the HyTest framework, the authors propose HyTestTF—a novel approach that integrates TReVa for validation of test results and incorporates FlaRe to introduce a hybrid model. This method enables efficient identification of flaky test cases and their triggering conditions with only one additional round of testing during early development stages. HyTestTF represents the first technique to automate flakiness detection in CPS testing at an early phase, substantially enhancing both reliability and efficiency. Empirical evaluation demonstrates that HyTestTF effectively distinguishes between flaky and non-flaky test cases, thereby improving the overall trustworthiness of CPS test results.

Abstract

Cyber-Physical Systems (CPSs) have many applications, ranging from simple thermostat systems to autonomous driving systems and medical devices. Like all systems, they need to function correctly. To this end, validation techniques such as testing that can effectively reveal faults are required. Also, CPSs usually operate in uncertain environments where various expected/unexpected events can affect their behaviors. These events together with timing and synchronization incidents may result in various/different CPS behaviors and may cause a CPS to pass a test under some conditions and fail it under others. Such test cases are called ``flaky'' test cases and we call the conditions that are responsible for them ``flaky conditions'' and call these behaviors ``flaky behaviors''. When test cases are flaky, testing results are unreliable. To achieve more reliable test results, engineers may attempt to recognize flaky test cases and remove them. In this work, beginning with a test case generation and execution technique called HyTest that we created previously, we integrate TReVa, a new technique that validates testing results provided by HyTest and FlaRe, a new technique that recognizes flaky test cases and flaky conditions using hybrid models during the early stages of CPS development in just one additional round of testing. We present the results of an empirical study evaluating the effectiveness of our new approach (which we call HyTestTF). Our results show that correctly differentiate flaky test cases from non-flaky test cases

Problem

• flaky test
• Cyber-Physical Systems
• test reliability
• flaky behavior
• uncertain environments

Innovation

• flaky test recognition
• hybrid models
• Cyber-Physical Systems (CPSs)
• test validation
• uncertain environments