State Field Coverage: A Metric for Oracle Quality

2025-10-03

AI Summary

Existing test oracle quality evaluation metrics suffer from poor generality and weak guidance, hindering effective oracle improvement. Method: We propose a novel metric—*state field coverage*—which identifies, via static analysis, the set of object fields accessed by test assertions and computes their proportion relative to all accessible fields in the class, thereby quantifying the breadth of program state examined by the oracle; its effectiveness is validated through mutation testing. Contribution: This work introduces the first general-purpose oracle quality metric that is strongly correlated with defect-detection capability, statically computable, and applicable to diverse assertion types—including invariants. Large-scale experiments across 273 invariants and 249,027 assertions demonstrate a high correlation (ρ > 0.8) between this metric and mutation score, confirming its validity. Crucially, it provides actionable, static guidance for oracle design and optimization.

Technology Category

• Search and Optimization: Evaluation and Analysis
• Machine Learning: Evaluation and Analysis
• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Security and Privacy: Large-scale security measurements
• User Modeling, Personalization and Recommendation: Metrics for user behavior and evaluating success

Abstract

The effectiveness of testing in uncovering software defects depends not only on the characteristics of the test inputs and how thoroughly they exercise the software, but also on the quality of the oracles used to determine whether the software behaves as expected. Therefore, assessing the quality of oracles is crucial to improve the overall effectiveness of the testing process. Existing metrics have been used for this purpose, but they either fail to provide a comprehensive basis for guiding oracle improvement, or they are tailored to specific types of oracles, thus limiting their generality. In this paper, we introduce state field coverage, a novel metric for assessing oracle quality. This metric measures the proportion of an object's state, as statically defined by its class fields, that an oracle may access during test execution. The main intuition of our metric is that oracles with a higher state field coverage are more likely to detect faults in the software under analysis, as they inspect a larger portion of the object states to determine whether tests pass or not. We implement a mechanism to statically compute the state field coverage metric. Being statically computed, the metric is efficient and provides direct guidance for improving test oracles by identifying state fields that remain unexamined. We evaluate state field coverage through experiments involving 273 representation invariants and 249,027 test assertions. The results show that state field coverage is a well-suited metric for assessing oracle quality, as it strongly correlates with the oracles' fault-detection ability, measured by mutation score.

Problem

• Assessing oracle quality to improve software testing effectiveness
• Introducing state field coverage metric for comprehensive oracle evaluation
• Providing static computation mechanism for efficient oracle quality guidance

Innovation

• State field coverage measures oracle's accessed object state proportion
• Static computation enables efficient identification of unexamined fields
• Metric correlates strongly with mutation score for fault detection