code templating

To address the challenges of costly full-repository builds and inefficient execution feedback in repository-level code generation, this paper proposes a sandbox-testing-driven paradigm for lightweight execution environment construction. Our method isolates the target function together with its minimal dependency set, enabling dynamic execution within an isolated sandbox to obtain precise, fine-grained feedback—bypassing the scalability bottlenecks of full-repository compilation. Key components include dependency-aware minimal extraction, automated test script generation, and construction of a large-scale function-level benchmark (RepoST-Train with 7,415 functions and RepoST-Eval). Experiments demonstrate substantial improvements in code model performance: Pass@1 increases by 5.5% on HumanEval and 3.5% on RepoEval. We further conduct systematic evaluation across 12 mainstream models. The proposed infrastructure enables highly scalable, low-coupling execution feedback for repository-level code generation.

This study addresses the lack of systematic understanding regarding the application domains, maintenance characteristics, and effective design practices of GitHub template repositories. Conducting the first large-scale empirical investigation, the work integrates data mining, statistical analysis, code quality assessment tools—detecting code smells, vulnerabilities, and security hotspots—and an LLM-as-a-judge classification approach to systematically uncover domain distributions, language-specific quality variations, and maintenance patterns. The findings reveal web development as the dominant application domain, with high-quality templates consistently adhering to software engineering best practices and offering comprehensive documentation. Through qualitative evaluation, the study distills actionable design guidelines and identifies common pitfalls, providing practical guidance for developers creating or using template repositories.

This study addresses the challenges of high cost, error-proneness, and defect propagation in cross-repository code and test reuse during software refactoring. Through action research, the authors conduct bidirectional empirical analyses on real-world cases such as Soot/SootUp and FindBugs/SpotBugs, identifying for the first time the bidirectional reuse requirements and semantic reuse patterns inherent in refactoring scenarios. They propose a semantic alignment–based code mapping approach coupled with a hierarchical, extensible clone detection mechanism. Experimental results demonstrate that their method reduces irrelevant clones by 33%–99% on average and achieves a benchmark precision of 86%. The practical impact is further evidenced by five reported issues and ten pull requests submitted to open-source communities, eight of which have already been merged, confirming the approach’s effectiveness and applicability.

This study systematically evaluates large language models—particularly GPT-4—for idiomatic Python code refactoring to enhance clarity, efficiency, and readability. We propose a prompt-engineering–based automated refactoring recommendation method that jointly identifies non-idiomatic patterns and generates idiomatic alternatives. Evaluation combines human assessment with static analysis tools (e.g., Pylint, Vulture) to benchmark accuracy, coverage, and contextual adaptability. Our empirical evaluation demonstrates, for the first time, that GPT-4 significantly outperforms traditional static analysis baselines in both recommendation accuracy and scope—especially in semantically nuanced, context-dependent refactoring scenarios requiring deep program understanding. A randomized human validation sample yields a 92.3% correctness rate for GPT-4’s suggestions, confirming its viability as a high-precision, context-aware assistant for idiomatic refactoring. The work establishes LLMs as robust, adaptive tools for practical, semantics-driven code improvement.

This study addresses the challenge of LLM-driven automated code quality assessment and improvement. We propose CodeQUEST, a framework featuring a closed-loop architecture with integrated evaluation and optimization modules. It enables, for the first time, fine-grained, multi-dimensional, and verifiable iterative optimization of code by an LLM (GPT-4o) across ten dimensions—including readability, maintainability, efficiency, and security. To ensure objectivity and reproducibility, we introduce a proxy-metric calibration mechanism that integrates established static analyzers (Pylint, Radon, Bandit) with multi-dimensional quantitative scoring. Experiments on Python and JavaScript benchmarks demonstrate an average relative quality improvement of 52.6%. Moreover, CodeQUEST’s assessments exhibit strong correlation (ρ > 0.89) with mainstream static analysis tools. The framework establishes a scalable, empirically grounded paradigm for integrating LLMs into software engineering practice.

Current evaluations of code generation predominantly rely on single-language benchmarks and unit tests, which inadequately capture the full spectrum of model capabilities. This work proposes PROBE—a scalable, multidimensional evaluation framework that systematically integrates five programming languages, multiple prompting strategies, and diverse metrics to holistically assess six prominent large language models across three key dimensions: functional correctness, solution proximity, and code quality. Leveraging functional testing, semantic similarity analysis, and static code quality assessment, the study reveals that while models perform well on simple tasks and in high-resource languages, they often produce unreliable outputs on complex problems or in low-resource languages due to fundamental errors, thereby exposing critical limitations in current code generation capabilities.

Existing benchmarks struggle to evaluate large language models’ ability to adapt code in the absence of explicit instructions, across multiple change types, and at the fragment-level granularity. This work proposes a mutation-injection framework based on open-source Java code, introducing— for the first time at the fragment level—a taxonomy of adaptation operations inspired by real developer behaviors. By leveraging controlled mutations and reinserting test suites, the framework assesses models’ contextual adaptation capabilities without requiring edit instructions. The approach supports multi-granular context control, enabling quantitative analysis of how different adaptation types affect model performance and revealing fundamental limits in scalability with respect to code complexity and contextual dependency.

This work addresses the challenge that existing code question-answering benchmarks struggle to distinguish whether models genuinely understand code logic or merely rely on memorized documentation or superficial patterns from pretraining. To this end, the authors propose an automated framework featuring an “answer-first” task generation mechanism and a three-condition evaluation paradigm—closed-book, code-only, and with documentation—to construct the first repository-scale code QA benchmark that explicitly disentangles code reasoning from document memorization. The framework employs tool-augmented agents to explore source code and generate verifiable questions, with responses evaluated by large language model judges along three dimensions: accuracy, completeness, and specificity. Experiments across 10 Python repositories yield 628 tasks, revealing that code access is the primary driver of performance gains (+0.23), documentation provides only marginal improvement (+0.071), and in code-derivable tasks, the code-only condition nearly matches full-documentation performance.

Template engine applications are notoriously difficult to debug and repair due to characteristics such as mixed-language composition, opaque data flows, and delayed validation, yet research in this area has long been scarce. This work presents the first large-scale empirical study of 1,004 real-world defects across 15 widely used template engines, systematically characterizing typical symptoms—predominantly abnormal rendering (48.61%)—identifying 17 root cause categories, and revealing collaborative repair patterns spanning both templates and host code (67.92% of fixes confined to templates, while over 20% require modifications to host logic). Based on these findings, the study offers actionable recommendations for developers and tool designers and implements two prototype debugging tools for the Jinja engine, demonstrably enhancing development and debugging efficiency for template-based applications.

Existing benchmarks struggle to evaluate large language models’ ability to preserve functional correctness during multi-turn code optimization dialogues. This work proposes the first systematic evaluation framework that generates such dialogues via a dynamic instruction selection algorithm and verifies functional consistency using test suites. Experimental results reveal a significant drop in functional correctness across mainstream large language models after multiple rounds of optimization, with accuracy rates falling between 19.2% and 69.2%. The degradation is most pronounced in scenarios involving logical refactoring and incorporation of new requirements, exposing a critical limitation in current models’ capacity to maintain code integrity through iterative modifications.