TestMap: Evidence Infrastructure for Foundation-Model-Assisted Test Generation

2026-06-08

AI Summary

This work addresses the lack of systematic validation for unit tests generated by foundation models, which hinders reliable assessment of their correctness, utility, and maintainability. To bridge this gap, we introduce TestMap—an open-source infrastructure tailored for C#/.NET projects—that establishes an evidence-centered framework for automated test generation across its entire lifecycle, encompassing mapping, execution, repair, evaluation, and experiment tracking. By integrating repository analysis, source-test mapping, coverage and mutation testing, static analysis, test smell detection, and model-guided generation, TestMap enables observable, reproducible, and comparable experimentation across diverse models, prompts, and strategies. The framework further uncovers limitations of current models, missing contextual information, repair overhead, and latent defects in the system under test, thereby providing an empirical foundation for trustworthy test generation.

Abstract

Foundation models (FMs) can generate plausible unit tests, but determining whether those tests are correct, useful, maintainable, and worth integrating remains difficult. Generated tests must be mapped to the code they target, inserted into real projects, built, executed, measured against the baseline suite, repaired when necessary, and compared across models and generation strategies. This validation process is fragmented across build systems, test runners, coverage tools, mutation tools, static analyzers, and experiment scripts. The problem is especially important because generated tests are both code artifacts and validation artifacts: they must themselves be validated before they can be trusted as evidence about the system under test. This paper presents TestMap, an open-source infrastructure prototype that automates evidence-backed foundation-model-assisted test generation for C#/.NET repositories. TestMap supports repository analysis, source-test mapping, baseline execution, code metric collection, test smell detection, coverage measurement, mutation testing, model-guided test generation, validation, repair, and repository-specific experiment tracking. Rather than reporting only final passing tests, TestMap records the lifecycle of each generated candidate, including failed, repaired, low-impact, and evidence positive outcomes. These intermediate outcomes can reveal model limitations, missing context, repair cost, toolchain inefficiencies, or possible faults in the system under test. Using TestMap as a design case, we describe the architecture and evidence model needed to make generated tests observable, repeatable, and comparable across repositories, models, prompts, and generation strategies. We conclude with lessons learned and open challenges, including oracle and assertion quality, metric attribution, test maintainability, flakiness, execution cost, and developer acceptance.

Problem

• foundation models
• test generation
• test validation
• evidence infrastructure
• software testing

Innovation

• foundation models
• test generation
• evidence infrastructure
• mutation testing
• test lifecycle