Semantic Drift in Bug Resolution: How Behavioral Signals Propagate from Reports to Tests and Patches

2026-07-20

AI Summary

This study addresses the challenge of behavioral semantic alignment among natural language bug reports, triggering tests, and code patches in software defect repair, arguing that lexical similarity alone is insufficient to capture behavioral consistency. To bridge this gap, the authors propose Desc2Fix, a novel framework that reformulates semantic alignment as measurable engineering signals by integrating structured behavioral anchors, multi-granularity embeddings (e.g., SBERT, CodeBERT, OpenAI), and large language models (GPT-4o, DeepSeek-Chat) for signal extraction and alignment scoring. Experiments on 2,857 defect triplets demonstrate that LLMs can extract behavioral signals with high completeness (up to 90%), and that full-code diffs provide a more reliable basis for alignment than localized code fragments, significantly enhancing the reliability of test generation and patch ranking.

Abstract

Bug resolution is a cross-artifact process: natural-language reports must expose actionable behavioral cues that can be reproduced by tests and corrected by patches. Yet the extent to which these signals are preserved across artifacts remains largely unquantified. We introduce Desc2Fix, a framework for measuring semantic alignment between bug reports, triggering tests, and developer-written fixes. Alignment is operationalized through structured behavioral anchors (e.g., reproduction steps, API/exception cues, expected vs. actual behavior), deterministic similarity metrics (ROUGE, SBERT, CodeBERT, OpenAI embeddings), and LLM-based judgments grounded in coverage, correctness, and specificity. We analyze 2,857 report-test-patch triplets from Defects4J and SWT-Bench using GPT-4o and DeepSeek-Chat. LLMs reliably extract structured signals (up to 90% completeness) and exhibit strong cross-model consistency, yielding a stable semantic input contract for downstream reasoning. However, alignment is highly representation-sensitive: lexical similarity alone is insufficient; full diffs provide the most stable basis for judging report-patch correspondence; and structured summaries trade surface similarity for predicate-level salience. Across more than 182,000 LLM alignment ratings, both models show systematic optimism relative to humans (1-2 points on 5-point scales) and only modest rank agreement, motivating bias-aware evaluation. Our results demonstrate that behavioral alignment is measurable but not reducible to similarity, and that structured anchors combined with embedding-based proxies provide reproducible signals for ranking and filtering tests and candidate patches. By transforming alignment into a controllable engineering signal, Desc2Fix enables more reliable test generation, semantics-aware fault localization, principled patch ranking, and improved bug report authoring.

Problem

• semantic drift
• bug resolution
• behavioral signals
• cross-artifact alignment
• bug reports

Innovation

• semantic alignment
• bug resolution
• structured behavioral anchors
• LLM-based evaluation
• Desc2Fix