The Impact of Configuring Agentic AI Coding Tools on Build-vs-Buy Decisions: A Study Protocol

2026-06-02

AI Summary

This study addresses the lack of systematic investigation into how AI programming agents decide between implementing functionality from scratch versus importing external libraries—a critical yet poorly understood aspect of code generation. To elucidate the behavioral mechanisms underlying these decisions, this work proposes a pre-registered, controlled experimental framework to systematically evaluate how contextual documentation, explicit prohibitions, skill definitions, MCP-based tool discovery, and permission controls influence library adoption choices in Claude Code and OpenAI Codex. Leveraging a staged benchmark suite and nine pre-registered hypotheses, the project will release the first public dataset and analysis pipeline to rigorously assess how these configurations affect agents’ tendencies toward library usage, as well as the completeness and accuracy of their disclosures regarding such choices.

Abstract

Agentic AI coding tools write code with increasing autonomy and in doing so decide when to import a library and when to implement functionality from scratch. These decisions, whether to build functionality from scratch or buy into an external library, hereafter build-versus-buy, carry direct consequences for software security, licensing compliance, performance, and long-term maintainability. Yet no controlled experimental study has examined what governs build-versus-buy decisions in agentic AI coding tools. Configuration mechanisms, i.e., the means by which developers tailor agentic AI coding tool behavior to a project or workflow, are one of the primary means by which practitioners can influence these decisions. However, it is unclear which configuration mechanisms influence build-versus-buy decisions most effectively. We present a pre-registered protocol to study how configuration mechanisms alter build-versus-buy behavior in two popular agentic AI coding tools: Claude Code and OpenAI Codex. We will execute controlled programming tasks drawn from a benchmark of staged projects, each constructed around identifiable build-versus-buy points, and will manipulate the configuration supplied to each tool, ranging from no configuration, through context files with soft preferences and explicit prohibitions, to Skills (instructions that can be autonomously discovered), MCP-enabled library discovery tools, and permission controls, measuring which libraries the tool selects, whether it discloses newly introduced libraries, and whether those disclosures are complete and accurate. Nine pre-registered hypotheses structure the protocol. The resulting benchmark dataset and analysis pipeline will be released as a reusable artifact for evaluating build-versus-buy behavior in agentic AI coding tools.

Problem

• Agentic AI
• build-versus-buy
• configuration mechanisms
• code generation
• software libraries

Innovation

• agentic AI coding tools
• build-versus-buy decisions
• configuration mechanisms
• controlled experiment
• dependency disclosure