Documentation vs. Code Patterns: What Drives LLM-Based Exception Oracle Generation?

2026-08-01

AI Summary

This study investigates whether large language models (LLMs) rely on explicit documentation—such as Javadoc @throws annotations—or implicit patterns in code and tests when generating exception-related test oracles. Through large-scale intervention experiments involving the removal of exception documentation and attribution-guided replacement ablations, the authors evaluate three oracle generation systems spanning different architectures and scales. The findings reveal, for the first time, that the high accuracy of current LLMs stems primarily from shortcut signals at the structural or lexical level rather than a deep semantic understanding of exceptions. Notably, oracle accuracy drops by less than 1% after removing @throws annotations, indicating that such documentation is not a critical basis for model predictions. These results challenge prevailing evaluation paradigms and underscore the necessity of causal analysis to uncover the true mechanisms underlying model decisions.

Abstract

LLM-based test oracle generation (TOG) methods report high accuracy on exception oracle generation, but it remains unclear what evidence drives these predictions. In particular, do models use explicit exceptional-behavior documentation such as Javadoc @throws clauses, or do they rely on recurring patterns in tests, code, and documentation? We investigate this question through a large-scale intervention-based study of three TOG systems spanning classifier-based and generative architectures and model sizes from roughly 110M to 7B parameters, evaluated on three real-world benchmarks comprising two generated-test datasets and a new benchmark of developer-written tests. We first remove Javadoc @throws clauses and find that accuracy changes only marginally, with the largest drop below one percentage point. This indicates that structured exception documentation is not the primary driver of exception-oracle prediction. We then apply attribution-guided substitution ablations to identify the signals that predictions depend on. The results show that high accuracy can be driven by shortcut signals: some models are highly sensitive to a small number of structural tokens, while others distribute reliance across many lexical cues. These findings challenge the assumption that strong exception-oracle accuracy reflects robust use of exception semantics. Future TOG systems should therefore be evaluated not only by whether they predict the correct oracle type, but also by whether their predictions are grounded in meaningful exception-triggering evidence.

Problem

• exception oracle generation
• LLM-based test oracle
• Javadoc @throws
• evidence reliance
• shortcut signals

Innovation

• exception oracle generation
• intervention-based study
• attribution-guided ablation
• shortcut signals
• test oracle generation