Reduction of Test Re-runs by Prioritizing Potential Order Dependent Flaky Tests

2025-10-30

AI Summary

To address result instability in automated testing caused by inter-test ordering dependencies, this paper proposes a fragility-aware test prioritization method based on shared static field characteristics. The approach employs static code analysis to identify shared static state within test classes that is prone to induce ordering dependencies, then constructs a lightweight fragility scoring model to prioritize high-risk tests—thereby accelerating defect exposure. This work is the first to use shared static fields as the core heuristic feature for ordering-dependent test prioritization. Evaluated across 27 modules, it achieves complete and accurate prioritized coverage of all ordering-dependent tests in 23 modules. On average, it reduces total test execution count by 65.92% and eliminates 72.19% of redundant re-executions, significantly improving detection efficiency while lowering computational resource overhead.

Technology Category

• Search and Optimization: Algorithm Configuration
• Reasoning under Uncertainty: Stochastic Optimization
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Security and Privacy: Large-scale security measurements
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization

Abstract

Flaky tests can make automated software testing unreliable due to their unpredictable behavior. These tests can pass or fail on the same code base on multiple runs. However, flaky tests often do not refer to any fault, even though they can cause the continuous integration (CI) pipeline to fail. A common type of flaky test is the order-dependent (OD) test. The outcome of an OD test depends on the order in which it is run with respect to other test cases. Several studies have explored the detection and repair of OD tests. However, their methods require re-runs of tests multiple times, that are not related to the order dependence. Hence, prioritizing potential OD tests is necessary to reduce the re-runs. In this paper, we propose a method to prioritize potential order-dependent tests. By analyzing shared static fields in test classes, we identify tests that are more likely to be order-dependent. In our experiment on 27 project modules, our method successfully prioritized all OD tests in 23 cases, reducing test executions by an average of 65.92% and unnecessary re-runs by 72.19%. These results demonstrate that our approach significantly improves the efficiency of OD test detection by lowering execution costs.

Problem

• Reducing test re-runs for order-dependent flaky tests
• Prioritizing tests using shared static field analysis
• Lowering execution costs in continuous integration pipelines

Innovation

• Prioritizing order-dependent tests to reduce reruns
• Analyzing shared static fields to identify flaky tests
• Reducing test executions and unnecessary reruns significantly