DryRUN: On the Role of Public Tests in LLM-Driven Code Generation

2026-04-23

AI Summary

This work addresses the overreliance of existing large language model (LLM)-based code generation approaches on publicly available test cases for validation—a critical limitation in real-world development scenarios where such test cases are often absent, leading to overfitting and overconfidence. To overcome this, we propose DryRUN, a novel multi-agent framework that, for the first time, enables fully test-case-free code generation. DryRUN leverages LLMs to autonomously generate inputs, simulate execution traces, and iteratively refine code through planning and self-correction mechanisms at the algorithmic level. Experiments on LiveCodeBench v6 (post-March 2025) demonstrate that DryRUN matches the performance of the current state-of-the-art method, CodeSIM, without using any public test cases, while significantly reducing output token consumption. This advancement markedly enhances the applicability and robustness of LLMs in practical software development environments.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Machine Learning: Large Multimodal Models (LMMs)
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• User Modeling, Personalization and Recommendation: Large Language Models (LLM) for user modeling and recommendation

Abstract

Multi-agent frameworks are widely used in autonomous code generation and have applications in complex algorithmic problem-solving. Recent work has addressed the challenge of generating functionally correct code by incorporating simulation-driven planning and debugging, where language models trace execution steps to verify logic. However, these approaches depend on human-provided public test cases to ground the debugging and simulation loop. Manually authoring comprehensive input-output examples is a labor-intensive bottleneck in the software development lifecycle. Because ground-truth input-output examples are rarely available prior to implementation in real-world software engineering, this dependency restricts methods to curated competitive programming benchmarks. Furthermore, we identify that reliance on these public tests induces an ``overconfidence gap,'' causing frameworks to overfit to simplistic examples and fail on hidden evaluations. In contrast, we observe that external sample inputs are not strictly necessary for code generation. We demonstrate that large language models can autonomously generate valid inputs and simulate execution traces to self-correct. Consequently, we develop DryRUN, a framework that eliminates the need for ground-truth samples by allowing the LLM to iteratively plan, autonomously generate its own inputs and simulate execution, mitigating algorithmic overconfidence. Evaluations on the LiveCodeBench v6 dataset (post-March 2025) demonstrate that DryRUN matches performance against CodeSIM, a state-of-the-art and public-test-dependent framework, while operating entirely without public test cases or external execution feedback while reducing output token consumption.

Problem

• public test cases
• code generation
• overconfidence gap
• software development
• execution simulation

Innovation

• DryRUN
• public-test-free
• autonomous input generation
• execution simulation
• LLM-driven code generation