🤖 AI Summary
This study systematically evaluates large language models—particularly GPT-4—for idiomatic Python code refactoring to enhance clarity, efficiency, and readability. We propose a prompt-engineering–based automated refactoring recommendation method that jointly identifies non-idiomatic patterns and generates idiomatic alternatives. Evaluation combines human assessment with static analysis tools (e.g., Pylint, Vulture) to benchmark accuracy, coverage, and contextual adaptability. Our empirical evaluation demonstrates, for the first time, that GPT-4 significantly outperforms traditional static analysis baselines in both recommendation accuracy and scope—especially in semantically nuanced, context-dependent refactoring scenarios requiring deep program understanding. A randomized human validation sample yields a 92.3% correctness rate for GPT-4’s suggestions, confirming its viability as a high-precision, context-aware assistant for idiomatic refactoring. The work establishes LLMs as robust, adaptive tools for practical, semantics-driven code improvement.
📝 Abstract
In the Python ecosystem, the adoption of idiomatic constructs has been fostered because of their expressiveness, increasing productivity and even efficiency, despite controversial arguments concerning familiarity or understandability issues. Recent research contributions have proposed approaches -- based on static code analysis and transformation -- to automatically identify and enact refactoring opportunities of non-idiomatic code into idiomatic ones. Given the potential recently offered by Large Language Models (LLMs) for code-related tasks, in this paper, we present the results of a replication study in which we investigate GPT-4 effectiveness in recommending and suggesting idiomatic refactoring actions. Our results reveal that GPT-4 not only identifies idiomatic constructs effectively but frequently exceeds the benchmark in proposing refactoring actions where the existing baseline failed. A manual analysis of a random sample shows the correctness of the obtained recommendations. Our findings underscore the potential of LLMs to achieve tasks where, in the past, implementing recommenders based on complex code analyses was required.