Predicting Developer Acceptance of AI-Generated Code Suggestions

2026-01-29

AI Summary

This study addresses the persistent issue in AI-powered programming tools where irrelevant or unaccepted code suggestions disrupt developer workflows, stemming from a lack of quantitative understanding of acceptance behavior. Leveraging a large-scale industrial dataset comprising 66,329 human-AI interaction logs, this work presents the first systematic analysis of the distinguishing characteristics between accepted and rejected code suggestions in real-world settings. Building upon empirical insights and feature engineering, the authors propose CSAP, a personalized, feature-driven model for predicting code suggestion acceptance. Evaluated on both imbalanced and balanced datasets, CSAP achieves accuracies of 0.973 and 0.922, respectively—representing up to a 140.1% improvement over baseline methods. The model substantially enhances the precision of filtering out ineffective suggestions, thereby significantly reducing unwarranted interruptions during software development.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Humans and AI: Human-Aware Planning and Behavior Prediction
• Reasoning under Uncertainty: Probabilistic Programming

Application Category

• User Modeling, Personalization and Recommendation: Fairness-aware retrieval and ranking
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions

Abstract

AI-assisted programming tools are widely adopted, yet their practical utility is often undermined by undesired suggestions that interrupt developer workflows and cause frustration. While existing research has explored developer-AI interactions when programming qualitatively, a significant gap remains in quantitative analysis of developers'acceptance of AI-generated code suggestions, partly because the necessary fine-grained interaction data is often proprietary. To bridge this gap, this paper conducts an empirical study using 66,329 industrial developer-AI interactions from a large technology company. We analyze features that are significantly different between accepted code suggestions and rejected ones. We find that accepted suggestions are characterized by significantly higher historical acceptance counts and ratios for both developers and projects, longer generation intervals, shorter preceding code context in the project, and older IDE versions. Based on these findings, we introduce CSAP (Code Suggestion Acceptance Prediction) to predict whether a developer will accept the code suggestion before it is displayed. Our evaluation of CSAP shows that it achieves the accuracy of 0.973 and 0.922 on imbalanced and balanced dataset respectively. Compared to a large language model baseline and an in-production industrial filter, CSAP relatively improves the accuracy by 12.6\% and 69.5\% on imbalanced dataset, and improves the accuracy by 87.0\% and 140.1\% on balanced dataset. Our results demonstrate that targeted personalization is a powerful approach for filtering out code suggestions with predicted rejection and reduce developer interruption. To the best of our knowledge, it is the first quantitative study of code suggestion acceptance on large-scale industrial data, and this work also sheds light on an important research direction of AI-assisted programming.

Problem

• AI-generated code suggestions
• developer acceptance
• AI-assisted programming
• code suggestion filtering
• developer-AI interaction

Innovation

• Code Suggestion Acceptance Prediction
• AI-assisted programming
• Developer-AI interaction
• Personalized filtering
• Empirical study