🤖 AI Summary
This study investigates the causal impact of adopting agent-based AI programming tools on software architecture quality, with a focus on changes in architectural smell density. Leveraging a dataset of 151 Java open-source repositories, it extends causal inference frameworks from the code level to the architectural level for the first time. The analysis integrates the Arcan tool for smell detection, a staggered difference-in-differences design, the Borusyak imputation estimator, and multiple robustness checks. Results indicate that while the total number of architectural smells shows no significant change post-adoption (+1.1%, p=0.82), code volume increases substantially (+12.8%, p=0.003), leading to a 6.7% reduction in smell density (p=0.004). This apparent improvement stems from denominator inflation—i.e., increased code size—rather than genuine architectural enhancement.
📝 Abstract
AI coding tools are now used by a majority of developers, and agentic use of these tools has popularized the practice colloquially called "vibe coding". Yet causal evidence on their effect on software architecture is scarce. Prior causal work has measured code-level outcomes (complexity, static analysis warnings); whether such degradation propagates to architecture-level outcomes remains unknown. We mine 151 open-source Java repositories, 74 with detectable agentic AI adoption (identified via configuration files and Co-Authored-By commit trailers) and 77 propensity-matched controls, across a 13-month per-repository window yielding 1,811 monthly Arcan snapshots. We estimate the causal effect of adoption on architectural smell density (ASD) with a staggered difference-in-differences design and the Borusyak imputation estimator, applying a causal design recently used for code-level metrics to the architecture level. Total smell counts are essentially unchanged (+1.1%, p = 0.82) while lines of code grow +12.8% (p = 0.003); the resulting 6.7% ASD decline (p = 0.004) is therefore a denominator effect rather than an architectural improvement. Per-type estimates and robustness checks (wild cluster bootstrap, Lee bounds, stale-observation sensitivity) corroborate the pattern; pre-trends are flat (Wald p = 0.90), consistent with parallel trends. Density-normalized outcomes can mislead when treatment affects system size: raw counts and explicit decomposition are required for causal mining studies of AI tool adoption. The complete replication package, including the curated 151-repository monthly panel, is publicly available.