Problem
Research questions and friction points this paper is trying to address.
code generation
LLM evaluation
benchmarking
software repository
coding agents
E2E-SWE: Benchmarking LLMs on Building Working Codebases from Scratch
🤖 AI Summary
This study addresses the lack of systematic reasoning evaluation and test independence for LLM-based code agents in repository-level generation by proposing an end-to-end benchmark. Agents are required to construct complete, installable software repositories from scratch using only natural language specifications, with functional correctness verified through hidden tests. Methodologically, this work introduces implementation-agnostic rigorous specification definitions and an iterative automated auditing-repair mechanism, integrating static analysis, trajectory examination, and multilingual automated verification to ensure task precision and feasibility. Experimental evaluations of thirteen state-of-the-art models yield pass@1 scores ranging from 11.7% to 67.7%, revealing distinct long-horizon planning patterns. Ultimately, this benchmark bridges the gap in full-repository generation evaluation and identifies promising directions for future optimization.
📝 Abstract
Coding agents powered by large language models (LLMs) are evolving from making localized code changes to developing complete software repositories. However, evaluating repository-scale generation remains challenging: tasks must demand system-level reasoning while ensuring that all evaluated behaviors are precisely specified and independent of any particular implementation. We introduce E2E-SWE, a benchmark for evaluating whether coding agents can build complete, functional software repositories end to end. E2E-SWE contains 186 whole-repository generation tasks spanning 11 programming languages. Given only a natural-language specification and an empty workspace, an agent must implement a complete, installable project that satisfies a comprehensive suite of hidden tests. Each task is constructed by a software engineer in collaboration with an LLM; together, they develop the test suite and a corresponding implementation-independent specification. To ensure that tasks are well specified and practically solvable, we further subject them to an iterative verification process in which autonomous agents audit and repair task defects using static inspection and failures observed from real model rollouts. Evaluating 13 frontier models, we find substantial variation in end-to-end repository generation ability, with pass@1 ranging from 11.7% to 67.7%, providing strong model differentiation while leaving considerable headroom for future progress. Analysis of agent trajectories further reveals long, front-loaded reasoning patterns, highlighting the planning and system-level reasoning required to construct working codebases from scratch.
Innovation
Methods, ideas, or system contributions that make the work stand out.
End-to-End Benchmark
Repository-Scale Generation
Implementation-Independent Specification
Iterative Verification
System-Level Reasoning
🔎 Similar Papers
No similar papers found.
