From Untestable to Testable: Metamorphic Testing in the Age of LLMs

2026-03-25

AI Summary

This work addresses the challenge of effectively validating large language model (LLM)-integrated systems, whose outputs are highly stochastic and often lack ground-truth annotations, rendering traditional testing methods inadequate. To overcome this limitation, the paper proposes an unsupervised validation framework based on metamorphic testing, which circumvents the need for explicit test oracles by establishing metamorphic relations between input transformations and corresponding output behaviors. By systematically applying metamorphic testing to LLM-augmented software, the approach significantly enhances testability and reliability in annotation-scarce scenarios. The proposed framework offers a scalable and practical paradigm for ensuring the quality of complex AI-driven systems, where conventional oracle-based verification is infeasible.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: (Large) Language Models
• Search and Optimization: Metareasoning and Metaheuristics

Application Category

• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

This article discusses the challenges of testing software systems with increasingly integrated AI and LLM functionalities. LLMs are powerful but unreliable, and labeled ground truth for testing rarely scales. Metamorphic Testing solves this by turning relations among multiple test executions into executable test oracles.

Problem

• Metamorphic Testing
• Large Language Models
• Software Testing
• Test Oracles
• AI Integration

Innovation

• Metamorphic Testing
• Large Language Models
• Test Oracles
• AI Software Testing
• Untestable Systems