coverage measurement

Existing retrieval evaluation metrics, such as precision and recall, struggle to assess the breadth of information coverage in retrieved results, particularly in retrieval-augmented generation (RAG) scenarios where it is critical to capture diverse key information. To address this limitation, this work introduces CoverageBench, the first cross-task, multi-domain benchmark specifically designed for evaluating information coverage. CoverageBench integrates topics, fine-grained information nuggets, relevance labels, and baseline rankings, moving beyond traditional document-level relevance paradigms. Released via Hugging Face Datasets, the benchmark enables reproducible, quantitative evaluation of the diversity and comprehensiveness of retrieved information, establishing a standardized platform for advancing research on information coverage in retrieval systems.

This work addresses a critical limitation in existing cloud skill testing, which focuses solely on task success rates and fails to reveal uncovered behaviors, leaving test adequacy unquantifiable. To bridge this gap, the study introduces the first formal definition of test coverage units, coverage relationships, and a computational pipeline for cloud skills. It proposes a coverage evaluation method grounded in natural language skill packages: by parsing user prompts and initial resource states, the approach reconstructs operational obligations and models workflow context to establish an end-to-end measurement pipeline. Integrating model-assisted candidate generation with expert review, the framework produces auditable coverage reports and closes the loop by mapping coverage gaps to source-level test improvement recommendations. Empirical results demonstrate that this methodology enables quantitative assessment of test coverage for production-grade cloud skills, substantially enhancing their reliability and observability.

Existing research often relies on code coverage and mutation score as proxy metrics to evaluate the effectiveness of test cases generated by large language models (LLMs), yet their correlation with actual fault-detection capability remains unclear. This study conducts a large-scale empirical analysis of test suites produced by diverse LLMs across varied testing scenarios, systematically examining the relationships among coverage, mutation score, and real defect detection performance. The findings reveal that the validity of these proxy metrics is highly context-dependent: they offer some predictive value in regression testing but prove unreliable when the target code contains faults. Furthermore, the size of the test suite has limited influence on these correlations. These results challenge conventional assumptions in test evaluation and provide new empirical grounding for assessing LLM-generated tests.

Widespread train-test overlap—often unreported—undermines the credibility and interpretability of language model evaluations. Method: We conduct the first systematic audit of disclosure practices across 30 major model developers, combining empirical data auditing, policy analysis, and developer interviews. Contribution/Results: Only 9 (30%) disclose overlap information—4 via open training data, 5 via statistics or methodology. Based on these findings, we propose a transparency reporting standard mandating public disclosure of either overlap statistics or representative training-data subsets. Our advocacy directly prompted three organizations to adopt critical new disclosures. This work has catalyzed community consensus on trustworthy evaluation benchmarks and established a methodological foundation and practical pathway for reproducible, verifiable large language model assessment.

Current test-driven development (TDD) practices lack demand-oriented, automated test case generation methods. Method: This paper proposes the first TDD-oriented text-to-test generation paradigm, leveraging domain-adaptive fine-tuning and structured prompt engineering to enhance GPT-3.5 for directly generating executable test cases from natural language requirements. Contribution/Results: We introduce a dedicated requirement–test alignment evaluation framework and empirically demonstrate the synergistic benefits of fine-tuning and prompt design. Evaluated on five large open-source projects, our approach generated 7,000 test cases achieving 78.5% syntactic correctness, 67.1% requirement alignment, and 61.7% code coverage—substantially outperforming existing baseline models.

Traditional test adequacy metrics, such as code coverage and mutation testing, focus on implementation details and struggle to capture discrepancies between expected and actual program behavior. This work proposes an automated approach that extracts method-level expected behaviors from natural language documentation and source code, then maps them to existing test cases, thereby formalizing and empirically evaluating “behavioral gaps”—a dimension of test adequacy independent of structural metrics. By integrating natural language processing, static analysis, and behavioral mapping techniques, our method identifies 20,729 behaviors across ten Java open-source libraries with 93.1% precision, revealing that 17.5% of expected behaviors remain entirely untested. Notably, these gaps persist even in methods exhibiting high code coverage or high mutation kill rates, exposing a systematic deficiency in current testing practices—including automatically generated tests—in validating intended program behavior.

This work addresses the limitations of existing RAG system evaluations, which rely on query-level annotations or reference answers and thus fail to comprehensively assess the behavioral coverage of retrieval components across test suites. To overcome this, we propose Chunk Coverage (CC), a novel metric that—without requiring test oracles—quantifies retrieval coverage by measuring the proportion of corpus chunks retrieved at least once during testing, thereby characterizing coverage at the structural level. Building upon CC, we develop an unsupervised framework for evaluating retrieval behavior and guiding test case selection and prioritization. Empirical results in clinical and financial domains demonstrate that CC-guided testing achieves 50% coverage 1.7× faster than random strategies and 4.2× faster than redundancy-based approaches, while improving fault detection capability (measured by APFD) by 10%–25%.

This work addresses a critical limitation in current RAG system evaluation, which relies on static snapshots and fails to capture errors induced by knowledge base evolution. The authors propose the first metamorphic testing framework tailored for RAG systems, formally defining failure types under knowledge changes and introducing 11 mutation operators operating at both indexing and retrieval context levels. They develop a high-precision metamorphic oracle that integrates pre- and post-chunking mutations, LLM-based reranking, and generator upgrades. Evaluated across five datasets with over 28,000 mutated samples, the approach reveals metamorphic violation rates of 4.9–10.2% and achieves oracle F1 scores ranging from 0.927 to 1.000—substantially outperforming RAGAS, whose F1 score peaks at 0.570.

This work addresses the significant heterogeneity in testing strategies and organization across open-source projects, which hinders rapid comprehension of their testing practices due to the lack of unified analytical tools. To bridge this gap, we propose TestMiner—a multilingual test practice analysis tool supporting languages such as Python, Java, Go, and Rust. By leveraging static code analysis and metadata extraction, TestMiner generates multidimensional visualizations encompassing test statistics, distribution, evolution, and dependency relationships. As the first tool to systematically enable cross-language and cross-ecosystem exploration of testing practices, TestMiner has been integrated into a software testing course, where it demonstrated marked effectiveness among 50 undergraduate students by significantly enhancing their understanding and critical analysis of core testing concepts, including test organization, evolution, mocking, and boundary testing.

Current technology-assisted systematic review stopping mechanisms predominantly target a predefined recall rate, often overlooking the sufficiency of information required for robust decision-making. This work proposes two confidence-based heuristic stopping strategies that dynamically assess whether the screened literature provides adequate evidence to support reliable conclusions, thereby replacing conventional recall-oriented approaches. Evaluated on standard datasets of diagnostic test accuracy systematic reviews, the proposed methods enable efficient dynamic stopping, substantially reducing the number of articles requiring screening while, in most cases, still yielding conclusions consistent with those derived from the full evidence set—thus better aligning with the practical demands of evidence-based decision-making.