Metamorphic Testing of Transpilers via Mutation Consistency of Programs

2026-07-31

AI Summary

Existing compiler testing techniques are often ill-suited for transpilers, as they typically lack multiple equivalent implementations and may produce non-executable output code. This work introduces metamorphic testing to transpiler validation by proposing the notion of “mutation consistency”: it defines metamorphic relations at the source-code level to verify whether structurally consistent and expected changes manifest in the generated code when the input DSL program undergoes semantics-preserving mutations. This approach enables defect detection without requiring execution of the generated code. We develop a mutation-based modeling method for metamorphic relations, a source-level structural consistency analysis mechanism, and implement an automated tool, MCP-Tester. Evaluated on real-world technology migration cases, our method effectively uncovers transpiler bugs that elude conventional fuzzing approaches.

Abstract

Transpilers are increasingly used for software development, especially in industrial domains that rely on domain-specific languages (DSLs), to allow engineers to work with familiar concepts and appropriate abstractions. Ensuring the correctness of these instruments is therefore critical in many industrial settings. This paper observes that existing approaches for compiler testing hardly generalize to transpilers. Differential testing approaches are hindered as multiple equivalent implementations of the transpiler under test are seldom available in practice. The approaches based on metamorphic testing assume the ability to execute the compiled binaries, an assumption that cannot be always made for transpilers, which oftentimes produce results expressed as source code, requiring complex toolchains, hardware-in-the-loop setups, and depending on non trivial inputs. This paper introduces a novel metamorphic testing technique tailored to transpilers. Instead of reasoning about the runtime behavior of compiled programs, our approach defines metamorphic relations directly over the source code produced by the transpiler. These relations capture a property that we call mutation consistency of the (transpiled) programs: mutation-style changes in the input DSL program must induce predictable and structurally consistent changes in the generated output. We implemented this idea in a tool, MCP-Tester, and evaluated it through a case study conducted in the context of a technology-transfer project. Our current empirical results indicate that the proposed approach can effectively reveal faults that would remain undetected with pure fuzzing.

Problem

• transpilers
• metamorphic testing
• mutation consistency
• compiler testing
• domain-specific languages

Innovation

• metamorphic testing
• transpilers
• mutation consistency
• domain-specific languages
• compiler testing