🤖 AI Summary
Existing Infrastructure-as-Code (IaC) defect taxonomies—originally designed for declarative, configuration-language-based tools like Puppet—are of uncertain applicability to programming-language-based IaC (PL-IaC) tools such as Pulumi, Terraform CDK, and AWS CDK. Method: We conducted a large-scale empirical study across open-source repositories and private enterprise codebases (including VTEX and Nubank), qualitatively coding 3,364 PL-IaC defect reports and augmenting classification with the ACID tool for automation and consistency. Contribution/Results: We empirically validate, for the first time, the broad applicability of the original eight-category taxonomy to PL-IaC. Configuration Data defects emerge as the most prevalent category; Idempotency and Security defects—though less frequent—are consistently observed across tools and organizations. This work extends the scope of IaC defect taxonomies, establishing a transferable theoretical framework and actionable foundation for PL-IaC quality assurance.
📝 Abstract
Background: As Infrastructure as Code (IaC) becomes standard practice, ensuring the reliability of IaC scripts is essential. Defect taxonomies are valuable tools for this, offering a common language for issues and enabling systematic tracking. A significant prior study developed such a taxonomy, but based it exclusively on the declarative language Puppet. It remained unknown whether this taxonomy applies to programming language-based IaC (PL-IaC) tools like Pulumi, Terraform CDK, and AWS CDK. Aim: We replicated this foundational work to assess the generalizability of the taxonomy across a broader and more diverse landscape. Method: We performed qualitative analysis on 3,364 defect-related commits from 285 open-source PL-IaC repositories (PIPr dataset) to derive a PL-IaC-specific defect taxonomy. We then enhanced the ACID tool, originally developed for the prior study, to automatically classify and analyze defect distributions across an expanded dataset-447 open-source repositories and 94 proprietary projects from VTEX (e-commerce) and Nubank (financial). Results: Our research confirmed the same eight defect categories identified in the original study, with idempotency and security defects appearing infrequently but persistently across projects. Configuration Data defects maintain high frequency in both open-source and proprietary codebases. Conclusions: Our replication supports the generalizability of the original taxonomy, suggesting IaC development challenges surpass organizational boundaries. Configuration Data defects emerge as a persistent high-frequency problem, while idempotency and security defects remain important concerns despite lower frequency. These patterns appear consistent across open-source and proprietary projects, indicating they are fundamental to the IaC paradigm itself, transcending specific tools or project types.