A Large-scale Class-level Benchmark Dataset for Code Generation with LLMs

2025-04-22

AI Summary

Existing code generation benchmarks predominantly focus on isolated functions, failing to assess models’ understanding and generation capabilities at the class level—critical for real-world software development. Method: We introduce Classeval, the first large-scale Python class-level benchmark, comprising over 842,000 authentic class skeletons (including signatures and docstrings) extracted from 13,174 open-source projects. Our pipeline leverages static parsing, structural extraction, and docstring alignment to preserve intra-class structure, dependencies, and contextual information. We propose the first systematic class-level evaluation framework, incorporating static code metrics for enhanced analyzability and validating the efficacy of class skeleton prompting. Evaluation employs ROUGE, BLEU, and TSED. Results: GPT-4 experiments yield mean scores of ROUGE-L = 0.80, BLEU = 0.59, and TSED = 0.73, demonstrating that high-fidelity class skeleton prompting substantially improves generation fidelity and structural correctness.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Evaluation and Analysis
• Search and Optimization: Evaluation and Analysis

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Recent advancements in large language models (LLMs) have demonstrated promising capabilities in code generation tasks. However, most existing benchmarks focus on isolated functions and fail to capture the complexity of real-world, class-level software structures. To address this gap, we introduce a large-scale, Python class-level dataset curated from $13{,}174$ real-world open-source projects. The dataset contains over 842,000 class skeletons, each including class and method signatures, along with associated docstrings when available. We preserve structural and contextual dependencies critical to realistic software development scenarios and enrich the dataset with static code metrics to support downstream analysis. To evaluate the usefulness of this dataset, we use extracted class skeletons as prompts for GPT-4 to generate full class implementations. Results show that the LLM-generated classes exhibit strong lexical and structural similarity to human-written counterparts, with average ROUGE@L, BLEU, and TSED scores of 0.80, 0.59, and 0.73, respectively. These findings confirm that well-structured prompts derived from real-world class skeletons significantly enhance LLM performance in class-level code generation. This dataset offers a valuable resource for benchmarking, training, and improving LLMs in realistic software engineering contexts.

Problem

• Addressing lack of class-level benchmarks for LLM code generation
• Providing real-world Python class dataset with structural dependencies
• Enhancing LLM performance via realistic class skeleton prompts

Innovation

• Large-scale Python class-level dataset from real projects
• Class skeletons with signatures and docstrings preserved
• GPT-4 generates classes with high similarity scores