CoSQA+: Pioneering the Multi-Choice Code Search Benchmark with Test-Driven Agents

2024-06-17

AI Summary

Existing code search benchmarks rely on subjective semantic annotations, lack functional validation and multiple-choice evaluation, and thus suffer from low accuracy and poor scalability. To address these limitations, we propose CoSQA+, the first high-quality benchmark for multiple-choice semantic code search. CoSQA+ introduces a novel test-driven agent annotation paradigm that integrates multi-LLM candidate generation, collaborative review by Python domain experts, multi-stage filtering, and automated unit testing for functional correctness verification. This approach achieves a 96.4% annotation accuracy—significantly surpassing the original CoSQA—and enables large-scale, reproducible, functionally grounded annotation. Models trained on CoSQA+ demonstrate consistent and substantial improvements across multiple-choice code retrieval tasks. All data, source code, and annotation procedures are publicly released.

Technology Category

• Natural Language Processing: Code Generation / Program Synthesis from Natural Language
• Search and Optimization: Sampling/Simulation-based Search
• Cognitive Modeling & Cognitive Systems: Agent Architectures

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Semantic code search, retrieving code that matches a given natural language query, is an important task to improve productivity in software engineering. Existing code search datasets face limitations: they rely on human annotators who assess code primarily through semantic understanding rather than functional verification, leading to potential inaccuracies and scalability issues. Additionally, current evaluation metrics often overlook the multi-choice nature of code search. This paper introduces CoSQA+, pairing high-quality queries from CoSQA with multiple suitable codes. We develop an automated pipeline featuring multiple model-based candidate selections and the novel test-driven agent annotation system. Among a single Large Language Model (LLM) annotator and Python expert annotators (without test-based verification), agents leverage test-based verification and achieve the highest accuracy of 96.4%. Through extensive experiments, CoSQA+ has demonstrated superior quality over CoSQA. Models trained on CoSQA+ exhibit improved performance. We provide the code and data at https://github.com/DeepSoftwareAnalytics/CoSQA_Plus.

Problem

• Improve semantic code search accuracy
• Address limitations in existing code datasets
• Introduce test-driven agent annotation system

Innovation

• Automated pipeline with model selections
• Test-driven agent annotation system
• Improved accuracy through test verification