Studying the Impact of Early Test Termination Due to Assertion Failure on Code Coverage and Spectrum-based Fault Localization

2025-04-06

AI Summary

This study systematically investigates, for the first time, the adverse impact of premature test termination—caused by early assertion failures—on code coverage and spectrum-based fault localization (SBFL) effectiveness. Empirical analysis of 2,387 failing test cases across 12 open-source projects reveals that 19.1% exhibit premature termination, significantly reducing line and branch coverage and degrading SBFL accuracy—evidenced by mean first ranking (MFR) increases of 15.1% and 10.7% for Ochiai and Tarantula, respectively. To address this, we propose and evaluate a lightweight assertion reordering technique that mitigates premature termination. After applying it, 55% of project versions show statistically significant coverage improvement, and SBFL’s average MFR improves by 12.9%. Our work uncovers a critical yet previously overlooked influence of test execution strategy on fault localization performance, offering both conceptual insight and practical methodology to enhance SBFL robustness.

Technology Category

• Search and Optimization: Evaluation and Analysis
• Knowledge Representation and Reasoning: Diagnosis and Abductive Reasoning
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

An assertion is commonly used to validate the expected programs behavior (e.g., if the returned value of a method equals an expected value) in software testing. Although it is a recommended practice to use only one assertion in a single test to avoid code smells (e.g., Assertion Roulette), it is common to have multiple assertions in a single test. One issue with tests that have multiple assertions is that when the test fails at an early assertion (not the last one), the test will terminate at that point, and the remaining testing code will not be executed. This, in turn, can potentially reduce the code coverage and the performance of techniques that rely on code coverage information (e.g., spectrum-based fault localization). We refer to such a scenario as early test termination. Understanding the impact of early test termination on test coverage is important for software testing and debugging, particularly for the techniques that rely on coverage information obtained from the testing. We conducted the first empirical study on early test termination due to assertion failure (i.e., early test termination) by investigating 207 versions of 6 open-source projects. We found that a nonnegligible portion of the failed tests (19.1%) is early terminated due to assertion failure. Our findings indicate that early test termination harms both code coverage and the effectiveness of spectrum-based fault localization. For instance, after eliminating early test termination, the line/branch coverage is improved in 55% of the studied versions, and improves the performance of two popular SBFL techniques Ochiai and Tarantula by 15.1% and 10.7% compared to the original setting (without eliminating early test termination) in terms of MFR, respectively.

Problem

• Impact of early test termination on code coverage
• Effect of assertion failure on fault localization
• Empirical study on early test termination in software testing

Innovation

• Investigates early test termination impact
• Empirical study on assertion failure effects
• Improves coverage and fault localization