PROGRESS: Property-Guided Regression Search for Semantic Falsification

2026-07-29

AI Summary

This work addresses the limitations of traditional regression testing, which often misclassifies bugs as expected behavior, and property-based testing, which struggles to effectively guide exploration toward deep program states and complex preconditions. The authors propose an intention-driven, coverage-guided evolutionary testing approach that synergistically combines code context with large language models to automatically generate executable properties exhibiting low implementation leakage. They introduce a property-aware fitness function that jointly optimizes structural coverage and semantic bug detection during evolutionary search. Implemented atop EvoSuite’s DynaMOSA algorithm, the jqwik framework, and a parameter binding mechanism, the method uncovered 328 real bugs (58% of all detected) across 25 large Java projects—significantly outperforming regression testing, which found none. Furthermore, it successfully satisfied all preconditions for 70 out of 150 challenging properties, far exceeding native jqwik’s 18.

Abstract

Search-based regression-test generation effectively explores complex program structures, yielding high structural coverage, but its oracles are derived from the system under test: faults already present are recorded as expected behavior rather than exposed. Property-based testing offers independent semantic oracles, but depends on high-quality properties and gives little guidance for reaching deep states or satisfying selective preconditions. We present PROGRESS (PROperty-Guided REgression Search for Semantic Falsification), integrating intent-driven properties into coverage-guided, search-based evolutionary test generation to reach deep program states and detect violations of intended behavior. PROGRESS (1) extracts intent-bearing code context and uses a language-model pipeline to generate executable jqwik properties while limiting implementation leakage; (2) extends EvoSuite's DynaMOSA with a search objective and property-aware fitness function per property, rewarding progress through preconditions and prioritizing falsifying executions; and (3) binds property parameters and uses jqwik-provided generators to connect quantified inputs to evolving test sequences, steering generation toward coverage and bug-detection goals. We evaluate PROGRESS on 25 large-scale Java systems against regression-test generation, standalone property-based testing, and context ablations. PROGRESS detects 328/562 current-version bugs (58%) versus none for regression-test generation, and satisfies all preconditions for 70/150 hard-to-reach properties versus 18 for standalone jqwik. Ablations show documentation and caller/callee context are key to generating valid executable properties. PROGRESS preserves structural exploration while exposing faults missed by regression-derived assertions; we release a comprehensive artifact package.

Problem

• regression testing
• property-based testing
• semantic falsification
• test oracle
• deep program states

Innovation

• property-based testing
• search-based testing
• semantic falsification
• evolutionary test generation
• language model