Methodological Harness in Agentic Software Engineering: An Empirical Study on Mining Software Repositories

2026-09-25

AI Summary

This study addresses the unclear practical adoption of agentic software engineering methodological frameworks within large-scale code repositories. To investigate this, we conduct the first empirical examination across 116,000 GitHub repositories, integrating mining software repositories techniques, stratified sampling, quantitative artifact detection, and qualitative coding analysis to systematically evaluate the prevalence, co-occurrence, and rule characteristics of eight coordination mechanisms. Our findings reveal an overall presence rate of at least one mechanism at 21.7%, rising to 65.3% among highly starred projects. However, fully integrated systems remain rare, as current practices are predominantly confined to the isolated application of individual mechanisms. This work provides foundational empirical evidence regarding how multi-agent coordination paradigms are currently operationalized in open-source software development, highlighting a significant gap between theoretical agentic frameworks and their holistic real-world implementation.

Abstract

Context. Agentic software engineering requires mechanisms to coordinate and govern agents'work. The framework motivating this study proposes a methodological harness with eight mechanisms: context engineering, persistent shared knowledge, executable specifications, N-version mindset and parallel agents, normative specifications, structured consultation, evidence-based acceptance, and graduated autonomy. These are realized through artifacts such as rule or context files, specifications, and architectural decision records, but the framework had not been empirically validated at repository scale. Objective. We analyze to what extent and how this harness is observable in repositories with agentic activity. RQ1 characterizes adoption through artifact prevalence, breadth, co-occurrence, and temporal evolution; RQ2 examines rule files - a key observable artifact and persistent source of agent instructions - to assess how their content reflects the proposed mechanisms. Method. From the AIDev dataset of 116,211 GitHub repositories, we analyzed 5,435 using a design stratified by visibility, measured through stars as an indicator of popularity and/or reputation. For RQ1, we detected and quantified artifacts and introduction dates for the seven observable mechanisms. For RQ2, we qualitatively coded rule files from 150 repositories. Results. Population prevalence of at least one mechanism is 21.7%, versus 65.3% among the most visible repositories. Multi-mechanism configurations are rare. Where rule files exist, they almost always guide the agent and state norms, while persistent shared knowledge, executable specifications, structured consultation, and graduated autonomy appear only in a minority. Conclusions. The harness is empirically observable, but mainly through isolated mechanisms rather than the integrated system proposed by the framework.

Problem

• Agentic Software Engineering
• Methodological Harness
• Mining Software Repositories
• Empirical Study
• Rule Files

Innovation

• Agentic Software Engineering
• Methodological Harness
• Mining Software Repositories
• Empirical Study
• Context Engineering