Learning from 53.6K Real-World Developer Edits of AI-Generated Code

2026-07-27

AI Summary

This work addresses the limitation of current large language models for code, which are trained on incomplete Git commit data and lack high-quality corpora that authentically capture developers’ editing behaviors on AI-generated code. To bridge this gap, the authors introduce DECODE, a novel dataset comprising 53.6K fine-grained edit trajectories collected at scale from real-world IDE usage, covering Python, TypeScript, and JavaScript. Analysis reveals that 31% of AI-suggested completions are ultimately deleted in full by developers. A 3B-parameter open-source model fine-tuned on DECODE significantly outperforms state-of-the-art, non-fine-tuned large models on code editing prediction tasks, demonstrating the critical value of developer-centric editing data for enhancing the performance of AI programming assistants.

Abstract

Imperfections in AI-generated code require that software developers modify the generated code manually, or by re-prompting an AI programming assistant. Manual code edits provide more realistic and granular information on editing behavior than Git commits, which only contain final successful code snippets. Yet, due to a lack of high-quality, realistic code editing data, LLMs are mostly trained on publicly available Git data (e.g., commits). To address this gap, we introduce DECODE (Developer Edits of Code Dataset), a dataset of 53.6K real-world in-IDE code edits of AI-generated code in Python, TypeScript, and JavaScript, sourced from 1K+ developers. First, we demonstrate the utility of DECODE for data analysis, obtaining insights on when, why, and how AI-generated code is edited. We find that most edits occur within the first 15 minutes after accepting an AI completion, resulting in the removal of AI completions in 31% of edit trajectories. Second, we use DECODE to benchmark the ability of LLMs to predict code edits. We find that finetuning on DECODE enables open-source 3B models to perform code edit prediction tasks significantly better than frontier LLMs. We then discuss implications of this work, emphasizing the necessity of developer-centric machine learning approaches for future AI programming assistants.

Problem

• AI-generated code
• code editing
• developer behavior
• large language models
• real-world data

Innovation

• real-world code edits
• AI-generated code
• developer-centric dataset
• code edit prediction
• in-IDE editing behavior