evaluation framework design

Large language models (LLMs) deployed for industrial test generation face critical reliability challenges due to rapid model iteration, leading to outdated evaluations and compromised production trustworthiness. Method: This paper introduces the first continuous evaluation framework for LLM-based test generation tailored to industrial settings. It pioneers a “continuous evaluation” paradigm integrating technical metrics (e.g., code coverage) with engineering metrics (e.g., maintainability, expert ratings), while systematically addressing real-world issues including data leakage and irreproducible results. The framework integrates industrial toolchains (e.g., SonarQube), supports dynamic test-case selection, robust prompt engineering, and auditable measurement infrastructure. Contribution/Results: A longitudinal empirical study at LKS Next demonstrates that the framework accurately tracks LLM capability evolution, identifies key bottlenecks impeding industrial deployment, and effectively enables trustworthy integration into DevSecOps pipelines.

Existing approaches struggle to effectively quantify the similarity and quality between synthetic and real data in evaluating tool-augmented agents. To address this gap, this work proposes SynAE, a novel framework that establishes the first multi-axis evaluation system tailored for multi-turn tool-use scenarios. SynAE introduces four fine-grained metric categories—assessing task instructions, tool invocations, final outputs, and downstream evaluation performance—to systematically measure synthetic data across dimensions of validity, fidelity, and diversity. Integrating natural language processing, trajectory modeling, and controllable generation techniques, the framework enables a reproducible evaluation pipeline and successfully identifies several representative failure modes in synthetic data generation. Empirical results demonstrate that such multidimensional assessment is essential for enhancing the reliability of agent evaluations.

Existing process mining benchmarks provide only macro-level performance metrics (e.g., throughput time, completion rate), hindering identification of concrete improvement opportunities. To address this, we propose an executable process execution benchmarking method: it aligns event logs from the target organization and benchmark processes based on behavioral similarity, automatically identifying semantically equivalent and substitutable activity units; then constructs a joint feasibility–performance-impact assessment framework to generate ranked, evidence-driven process modification recommendations. This work pioneers the shift from descriptive benchmark analysis to prescriptive, “actionable” improvement guidance. Evaluated across multiple real-world process scenarios, our approach achieves an average throughput time reduction of 12.7%, significantly enhancing both the precision and implementability of process optimization.

Current AI evaluation methodologies suffer from a lack of standardized, systematic documentation, severely undermining reproducibility, transparency, and trustworthy decision-making. Method: This paper introduces Eval Factsheets—a novel framework that pioneers the application of structured documentation to AI evaluation. It establishes a five-dimensional taxonomy—encompassing Context, Scope, Structure, Methodology, and Alignment—to uniformly characterize diverse evaluation paradigms, including traditional benchmarks and LLM-as-judge approaches. A taxonomy-guided questionnaire specifies mandatory and recommended fields covering the entire evaluation lifecycle. Contribution/Results: Empirical validation across multiple benchmark cases demonstrates that Eval Factsheets consistently represent heterogeneous evaluation practices, significantly enhancing cross-evaluation comparability, reproducibility, and transparency. The framework provides a foundational, extensible tool for standardizing AI evaluation documentation and practice.

To address the challenges of excessive experimental scale, high resource consumption, and the trade-off between accuracy and efficiency in system-level LLM inference performance evaluation (e.g., throughput, latency), this paper proposes FMwork—a framework for efficient and reliable benchmarking. FMwork establishes a controlled test environment, introduces meta-metrics to quantify the cost–accuracy trade-off, designs a parameter selection strategy grounded in hardware–software interaction characteristics, and formulates a joint cost–performance optimization model. It achieves 96.6% accuracy relative to full-scale testing with only minimal samples—e.g., just 128 output tokens for Llama 3.1 8B—while improving experimental efficiency by up to 24× and delivering an additional 2.7× inference acceleration. Its core contribution is the first introduction of a meta-metric-driven sparse evaluation paradigm for LLM inference benchmarking, significantly enhancing scalability and reliability in large-scale performance analysis.

This study addresses the ambiguity and inconsistency in evaluation criteria for software engineering replication studies, which have led to contradictory interpretations and uncertainty in reported results. Through a systematic review of ten replication studies published between 2021 and 2025, combined with qualitative content analysis, statistical principles, and modeling of measurement uncertainty, this work is the first to uncover the heterogeneity and lack of standardized practices in current evaluation approaches. Building on these insights, the paper proposes a unified evaluation framework that integrates statistical theory, methodological rigor, and measurement theory. Empirical illustration demonstrates that the framework effectively enhances the transparency, consistency, comparability, and reliability of replication studies in software engineering.

This study addresses a critical limitation of existing DORA metrics, which rely solely on first-order statistics and thus fail to capture the distributional characteristics of software release cadence or distinguish teams with markedly different release regularity. To overcome this, the work introduces second-order statistics into the DORA framework for the first time, proposing a novel Delivery Consistency (DC) metric based on the coefficient of variation of inter-release intervals. It further constructs an eight-prototype Delivery Health Matrix to enable multidimensional diagnosis and targeted intervention for software delivery rhythms across platforms. Validation using real-world data spanning 120 weeks from four platforms—including Jira, GitHub, and Firebase—demonstrates that the approach effectively identifies teams sharing identical DORA ratings yet exhibiting divergent release patterns, uncovering underlying organizational or process constraints common to such teams.