Method-level Change-proneness: A Better Metric for Black-box Test Suite Minimization

2026-05-13

AI Summary

Existing black-box test suite minimization approaches struggle to accurately identify test cases with high fault-detection capability due to limitations in the scalability and efficiency of their metrics. To address this, this work proposes a novel metric—method-level change-proneness—that refines the traditional class-level change-proneness down to the method level for the first time. The approach leverages version control metadata to compute method change frequency, analyzes dependencies via test call graphs, and employs statistical measures to score test cases for constructing minimized suites. Evaluated on 635 buggy versions across 15 open-source Java projects, the proposed method achieves an average precision of 0.93 and a fault-detection rate of 0.94, significantly outperforming both class-level change-proneness and similarity-based baselines.

Abstract

Test Suite Minimization (TSM) reduces the size of test suites while preserving their fault detection capability. In black-box TSM, reduction is performed without analyzing production code. While several black-box TSM approaches have explored metrics like test logs or test similarity, those often suffer from scalability and efficiency issues. On the other hand, change-proneness (CP), recently emerged as an efficient and scalable alternative metric, has only been applied at class level. To accurately identify fault-revealing test cases, we propose CP at finer-grained method-level and implement Method-level Change-proneness based Test-suite Minimization (MCTM). MCTM first calculates CP for each method from version control metadata, then determines the dependency between test cases and methods by analyzing the test-code call-graph. Next, it scores the association between test cases and their invoked methods using statistical measures such as Average, Geometric Mean etc. Finally, test cases with the highest scores are selected to form the reduced suite. Evaluation on 15 open-source Java projects with 635 buggy versions shows MCTM achieves 0.93 accuracy and 0.94 fault detection rate on average, significantly outperforming class-level CP and similarity-based approaches while maintaining superior efficiency.

Problem

• Test Suite Minimization
• Black-box Testing
• Change-proneness
• Fault Detection
• Method-level

Innovation

• method-level change-proneness
• test suite minimization
• black-box testing
• fault detection
• call graph analysis