Automating Code Review: A Systematic Literature Review

2025-03-12

AI Summary

This study addresses the time-consuming and inefficient nature of manual code review by conducting a structured systematic literature review (SLR) of 119 publications—the first to propose a comprehensive, task-dimensional taxonomy for automated code review. Methodologically, it integrates machine learning, information retrieval, program analysis, and natural language processing techniques, and empirically evaluates approaches using datasets from GitHub, Gerrit, and other platforms, with metrics including BLEU, F1, and MAP. Key contributions are: (1) a refined classification of 12 automated review tasks and 7 core technical paradigms; (2) a curated inventory of 32 publicly available tools and datasets; (3) identification of critical bottlenecks in data-driven methods—particularly regarding interpretability and cross-project generalizability; and (4) a reproducible evaluation benchmark alongside four concrete directions for future research. The findings are synthesized into a rigorous, structured SLR report.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Evaluation and Analysis
• Knowledge Representation and Reasoning: Automated Reasoning and Theorem Proving

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Web Mining and Content Analysis: Robustness and generalizability of Web mining methods

Abstract

Code Review consists in assessing the code written by teammates with the goal of increasing code quality. Empirical studies documented the benefits brought by such a practice that, however, has its cost to pay in terms of developers' time. For this reason, researchers have proposed techniques and tools to automate code review tasks such as the reviewers selection (i.e., identifying suitable reviewers for a given code change) or the actual review of a given change (i.e., recommending improvements to the contributor as a human reviewer would do). Given the substantial amount of papers recently published on the topic, it may be challenging for researchers and practitioners to get a complete overview of the state-of-the-art. We present a systematic literature review (SLR) featuring 119 papers concerning the automation of code review tasks. We provide: (i) a categorization of the code review tasks automated in the literature; (ii) an overview of the under-the-hood techniques used for the automation, including the datasets used for training data-driven techniques; (iii) publicly available techniques and datasets used for their evaluation, with a description of the evaluation metrics usually adopted for each task. The SLR is concluded by a discussion of the current limitations of the state-of-the-art, with insights for future research directions.

Problem

• Automating code review tasks to save developers' time.
• Categorizing automated code review tasks and techniques.
• Identifying limitations and future research in code review automation.

Innovation

• Automates reviewer selection using data-driven techniques
• Recommends code improvements via automated review tools
• Systematically reviews 119 papers on code review automation