Compact Documentation for Coding Agents: A Benchmark, an Optimizer, and Why It Does Not Transfer

2026-09-25

AI Summary

This study investigates whether natural language documentation enhances the ability of coding agents to solve software engineering tasks. We construct a round-trip benchmarking framework and propose a code-regeneration-based documentation fidelity scoring mechanism, revealing that document completeness is more critical than length. Additionally, we design a description optimizer to generate high-fidelity, general-purpose documentation. Through experiments integrating large language models with automated prompt engineering on real-world repository repair tasks, we demonstrate that although full-fidelity documentation generation is achieved, static documentation does not significantly improve problem-solving rates. These findings challenge the prevailing assumption that "more documentation is better," providing crucial empirical evidence for agent-oriented documentation engineering.

Abstract

We investigate whether natural-language documentation helps coding agents resolve software issues, and we build the tools to construct and evaluate it. We introduce a roundtrip benchmark that scores code descriptions by whether code regenerated from them passes the original tests, and show that completeness, not length, drives a description's fidelity. Using the benchmark as an optimization signal, we discover a description-writing prompt that reaches full fidelity and generalizes to unseen files. We then test the hypothesis that motivated the work: that better documentation helps an agent resolve real repository issues. Across two model families and ten repositories, and against a positive control confirming that our evaluation can detect a genuine improvement, we find that it does not. When the source is present, neither static compact documentation nor retrieved context beats the issue alone. We report this negative result together with the benchmark and the optimizer, and we characterize the boundary at which documentation helps.

Problem

• coding agents
• software documentation
• issue resolution
• benchmark
• negative result

Innovation

• coding agents
• roundtrip benchmark
• prompt optimizer
• compact documentation
• negative result