Are Production Cloud Skills Adequately Tested? Measuring and Governing Skill Test Coverage in Practice

2026-07-24

AI Summary

This work addresses a critical limitation in existing cloud skill testing, which focuses solely on task success rates and fails to reveal uncovered behaviors, leaving test adequacy unquantifiable. To bridge this gap, the study introduces the first formal definition of test coverage units, coverage relationships, and a computational pipeline for cloud skills. It proposes a coverage evaluation method grounded in natural language skill packages: by parsing user prompts and initial resource states, the approach reconstructs operational obligations and models workflow context to establish an end-to-end measurement pipeline. Integrating model-assisted candidate generation with expert review, the framework produces auditable coverage reports and closes the loop by mapping coverage gaps to source-level test improvement recommendations. Empirical results demonstrate that this methodology enables quantitative assessment of test coverage for production-grade cloud skills, substantially enhancing their reliability and observability.

Abstract

Cloud platforms increasingly deliver reusable Cloud Skills that guide AI agents in creating, configuring, monitoring, and troubleshooting cloud resources. Workflow-oriented Skills specify multiple resource operations, user choices, validation steps, and recovery behavior. Existing Skill evaluation primarily measures whether a Skill improves task success. Passing the available testcases, however, does not reveal which behaviors specified by the Skill have never been tested. This paper introduces Skill Test Coverage, which measures how completely a Skill's test suite covers its operational test obligations. We define the coverage units, testcase coverage relation, calculation procedure, and claim boundary. Because these units and links are not explicit in natural-language Skill packages, we develop a measurement pipeline that recovers operational obligations, organizes their workflow context, maps testcases using both user prompts and initial resource states, and produces an auditable coverage report. Model-assisted candidate generation is combined with expert review to ensure that the reported coverage remains grounded in the original Skill and testcase evidence. A post-review module then translates confirmed coverage gaps into source-grounded test-improvement recommendations.

Problem

• Cloud Skills
• Test Coverage
• AI Agents
• Workflow-oriented Skills
• Skill Evaluation

Innovation

• Skill Test Coverage
• Cloud Skills
• Test Obligations
• Coverage Measurement
• AI Agent Validation