DiffTestGen: Change-Directed LLM-Based Testing for Exposing Behavioral Differences

2026-07-17

AI Summary

This work addresses the challenge of effectively triggering behavioral discrepancies caused by code changes, which often leads to missed regression defects in existing testing approaches. The authors propose a novel LLM-driven differential testing method tailored to code modifications, which innovatively integrates static call graph analysis, project documentation understanding, and large language model (LLM) generation capabilities. Guided by a joint coverage feedback mechanism, the approach iteratively refines test case generation to simultaneously enhance coverage of changed code across both old and new software versions. Evaluated on 463 pull requests, the method successfully exposed behavioral differences in 78.2% of them, achieved an average joint coverage of 90.7%, uncovered 99 additional issues compared to baseline techniques, improved coverage by 12.5–15.6 percentage points, and effectively detected regression defects overlooked by prior methods.

Abstract

As software evolves over time, it is important to ensure that any behavioral changes occur as intended by developers. A promising approach for this goal is to generate tests that expose behavioral differences between the old and new versions of a program. However, current approaches fail to trigger behavioral differences for many code changes. This paper presents~DiffTestGen, a novel change-directed, LLM-based differential testing approach specifically designed to expose behavioral differences introduced by a code change. The approach is enabled by two key contributions: First, DiffTestGen leverages static call graph analysis and project documentation to identify valid entry points for test generation and to guide the LLM toward reaching the changed code. Second, DiffTestGen iteratively improves our newly introduced union coverage metric, which combines coverage of modified code in the old and the new version, by providing targeted coverage feedback to the LLM. We evaluate DiffTestGen on two datasets comprising a total of 463 PRs. DiffTestGen exposes behavioral differences in 78.2% of the PRs while achieving an average union coverage of 90.7%. Compared with the baselines, DiffTestGen exposes 99 more PRs overall and increases code coverage by 12.5% and 15.6% percentage points, respectively. By integrating DiffTestGen with the Testora regression detector, we show that the identified behavioral differences can be used to detect regression bugs missed by the best existing approaches.

Problem

• behavioral differences
• differential testing
• code changes
• regression detection
• test generation

Innovation

• change-directed testing
• LLM-based test generation
• differential testing
• union coverage
• behavioral difference detection