Explanation-Guided Metamorphic Testing of Specialized Language Models: An Empirical Study

2026-08-07

AI Summary

Existing approaches struggle to effectively evaluate the robustness of domain-specific language models under semantic-preserving transformations. This work proposes a novel framework that integrates interpretability-guided perturbation with metamorphic testing, systematically generating high-fidelity adversarial examples through attribution-based token prioritization, large language model–driven semantic-equivalent rewrites, and automated semantic validation. The method reveals, for the first time, the models’ reliance on shortcut cues such as named entities and formatting signals. Compared to heuristic baselines, it improves the efficiency of generating effective failure-inducing test cases by 2.3× while maintaining strong semantic validity and label consistency.

Abstract

\head{Background} Task-specialized language models are increasingly integrated into software engineering workflows to support vertical-domain activities such as issue triaging, document classification, and automated analysis. Despite their adoption, there is limited empirical evidence on how to test their robustness and detect brittle behaviors under semantics-preserving input transformations. \head{Aims} This paper investigates whether explainability-guided metamorphic testing can improve the effectiveness and validity of robustness testing for specialized language models compared to heuristic mutation strategies. \head{Method} We conduct a large-scale empirical study of explanation-guided metamorphic testing across three datasets, four model architectures, and 20 testing configurations derived from combinations of attribution methods and mutation strategies. The evaluated configurations combine attribution-based token prioritization, LLM-driven mutation, and automated semantic verification to generate linguistically valid test variants. We assess failure discovery capability, semantic validity, and testing efficiency against heuristic baselines. \head{Results} Explanation-guided metamorphic testing generates 2.30$\times$ more verified failure-inducing test cases than heuristic mutation strategies. Semantic verification substantially improves mutation validity and achieves high label-preservation precision among gate-accepted variants according to human annotation. The study further reveals systematic shortcut behaviors across models, including over-reliance on named entities and formatting cues. \head{Conclusions} The results provide evidence that explanation-guided metamorphic testing is an effective and practical approach for empirically evaluating the robustness of task-specialized language models used in vertical AI applications.

Problem

• specialized language models
• robustness testing
• metamorphic testing
• semantic-preserving transformations
• brittle behaviors

Innovation

• explanation-guided testing
• metamorphic testing
• attribution methods
• semantic validity
• specialized language models