Case2Code: Scalable Synthetic Data for Code Generation

2024-07-17

International Conference on Computational Linguistics

AI Summary

Scaling high-quality synthetic data generation for code large language models remains challenging—existing approaches rely on expensive, powerful teacher models and suffer from limited diversity and correctness. Method: This paper introduces the Case2Code task, the first scalable synthetic paradigm that frames program behavior induction (case-to-code) as an end-to-end pipeline: large language models generate diverse inputs; automated execution yields ground-truth outputs; and dynamic testing rigorously validates functional correctness—eliminating dependence on teacher models while enabling low-cost, high-diversity, high-fidelity code data synthesis. Contribution/Results: Empirical evaluation demonstrates that models trained on Case2Code data achieve significant improvements in both case-to-code generalization and standard benchmarks (HumanEval, MBPP), confirming the effectiveness and transferability of inductive synthetic data for code modeling.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Cognitive Modeling & Cognitive Systems: Computational Creativity

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactions
• Search and Retrieval-Augmented AI: Large language models for search

Abstract

Large Language Models (LLMs) have shown outstanding breakthroughs in code generation. Recent work improves code LLMs by training on synthetic data generated by some powerful LLMs, which can be challenging to scale due to the dependence on a teacher model and high generation costs. In this paper, we focus on synthesizing code data at scale and propose a extbf{Case2Code} task by exploiting the expressiveness and correctness of programs. extbf{Case2Code} is an inductive inference task that aims to infer underlying code implementations by observing input-output examples or program behaviors, By incorporating LLMs to generate program inputs, and executing the program with these inputs to obtain the program outputs, we can synthesize diverse and high-quality extbf{Case2Code} data at scale for training and evaluating code LLMs. Experimental results show that case-to-code induction is challenging for current representative LLMs if they are untrained. Models trained with extbf{Case2Code} improve performance not only on distribution case-to-code induction but also on various coding-generation tasks, demonstrating the great potential of large-scale synthetic data and inductive learning.

Problem

• Scalable synthetic data generation
• Code generation using LLMs
• Inductive inference in programming

Innovation

• Case2Code: scalable code synthesis
• LLMs generate diverse program inputs
• Inductive learning enhances code generation