diagnostic evaluation

Configuration space explosion complicates performance impact modeling, while gray-box approaches rely on structural knowledge (e.g., module execution graphs) to improve model accuracy—yet the mechanisms by which structural features (e.g., number of modules or configuration options) and structural knowledge influence modeling difficulty and optimization potential remain unclear. Method: We formally define “modeling hardness” and “improvement opportunity,” establishing an analytical framework and matrix to quantify the interplay among system structural complexity, structural knowledge level, and modeling benefit. Controlled experiments on synthetic systems integrate module execution graph analysis with gray-box modeling. Contribution/Results: We identify module count and configuration option count as dominant determinants of modeling hardness. Under high hardness, strong structural knowledge significantly increases improvement opportunity. Structural knowledge primarily enhances ranking accuracy, whereas hardness predominantly degrades prediction accuracy. Our findings provide theoretical foundations and strategic guidance for allocating structural knowledge investment according to specific modeling objectives.

Large language models (LLMs) deployed for industrial test generation face critical reliability challenges due to rapid model iteration, leading to outdated evaluations and compromised production trustworthiness. Method: This paper introduces the first continuous evaluation framework for LLM-based test generation tailored to industrial settings. It pioneers a “continuous evaluation” paradigm integrating technical metrics (e.g., code coverage) with engineering metrics (e.g., maintainability, expert ratings), while systematically addressing real-world issues including data leakage and irreproducible results. The framework integrates industrial toolchains (e.g., SonarQube), supports dynamic test-case selection, robust prompt engineering, and auditable measurement infrastructure. Contribution/Results: A longitudinal empirical study at LKS Next demonstrates that the framework accurately tracks LLM capability evolution, identifies key bottlenecks impeding industrial deployment, and effectively enables trustworthy integration into DevSecOps pipelines.

Current evaluation methods for large language models (LLMs) primarily identify failing samples or categories but struggle to uncover underlying capability deficiencies, thereby limiting targeted model improvement. This work proposes CRAFT, a novel framework that diagnoses model weaknesses at the scoring-criterion level. CRAFT constructs a hierarchical capability tree by extracting capability descriptions and applying hierarchical clustering, then dynamically identifies low-performance nodes across multiple granularities to generate targeted fine-tuning data. Evaluated on financial and legal domains as well as 13 standard benchmarks, CRAFT significantly outperforms prompt-clustering and random data generation baselines. Fine-tuning four open-source LLMs with CRAFT-generated data consistently enhances their performance, demonstrating more precise localization of capability gaps and enabling efficient, targeted model refinement.

Fixed-size benchmarking in model evaluation often fails to balance efficiency, statistical reliability, and diverse objectives, leading to either excessive resource consumption or unreliable results. This work proposes the first adaptive framework that integrates sequential testing into AI model evaluation, dynamically allocating evaluation data based on stopping criteria tailored for model ranking and selection tasks. By combining sequential hypothesis testing, minimum detectable effect analysis, and diminishing returns detection, the method achieves substantial gains in efficiency without compromising rigor. Empirical validation on the Open VLM Leaderboard demonstrates an 80% reduction in computational cost while maintaining a confidence interval width of 2.5 points, significantly enhancing both the practicality and scalability of model evaluation.

This study addresses the uncontrolled quality of quality engineering (QE) artifacts—such as requirements specifications, test cases, and Behavior-Driven Development (BDD) scenarios—automatically generated by large language models (LLMs). We propose an iterative optimization framework integrating forward generation, backward generation, and rubric-guided scoring to enhance artifact quality along four dimensions: clarity, completeness, consistency, and testability. Our approach enables automated, quantitative, and reproducible quality assessment and improvement. Evaluated across 12 real-world projects, the method significantly improves output stability: it preserves high quality under high-quality inputs and substantially outperforms baselines under low-quality inputs. The core contribution is the first integration of backward generation with structured rubric-based guidance, establishing a closed-loop, artifact-centric quality enhancement paradigm for QE.

Traditional AI evaluation methods are primarily designed for static model selection and often fail to diagnose root causes of performance degradation in production or guide targeted improvements. This work proposes EvalLoop, a novel methodology that embeds evaluation into a continuous optimization loop. By integrating dimensional metric grouping, failure mode categorization, single-variable controlled experiments, and a human-in-the-loop gating mechanism, EvalLoop enables precise mapping from failure attribution to actionable refinements and supports deployment-aware model selection. Evaluated on a sales intelligence briefing generation task, the approach increased overall accuracy of the best-performing model from 82.6% to 94.6%, improved performance on critical dimensions by over 16 percentage points, and reduced human review effort by 94%.

Current benchmarks for evaluating toxicity in large language models exhibit underappreciated systematic biases that may lead to the deployment of unsafe models. This work systematically investigates how variations in task formulation—such as text completion versus summarization—input data domains, and evaluated models interact with multiple toxicity metrics. It reveals, for the first time, that both task type and data domain significantly influence toxicity scores. Experiments demonstrate that existing benchmarks are prone to misclassifying content as harmful when tasks are altered and show inconsistent performance across domains, highlighting their fragility and dependence on specific model-task configurations. These findings underscore the urgent need for more robust and reliable toxicity evaluation frameworks.

This study addresses the lack of systematic evaluation of existing test selection metrics under multi-objective settings, distribution shifts, and multimodal data—challenges that hinder practical metric selection. To bridge this gap, the authors construct the first unified benchmark encompassing three testing objectives (fault detection, performance estimation, and retraining guidance), five types of distribution shifts, three data modalities (images, text, and Android packages), and 13 deep learning models. Through a large-scale empirical study involving 1,640 experimental scenarios, they conduct rigorous statistical analyses to comprehensively compare the performance of 15 widely used metrics, elucidate their respective applicability boundaries, and provide reliable guidance and actionable recommendations for test selection in safety-critical systems.

Current evaluations of large language models (LLMs) on ill-defined tasks—such as complex instruction following and natural language-to-Mermaid sequence diagram generation—suffer from insufficient coverage, sensitivity to phrasing, incomparable metrics, and instability in LLM-based judging, thereby failing to yield reliable or diagnostic assessment signals. This work presents the first systematic analysis of confounding failure modes in such tasks, integrating case studies, failure mode categorization, and a multidimensional evaluation framework to demonstrate how existing benchmarks often conflate distinct error types, leading to distorted scores. Moving beyond monolithic aggregate metrics, the proposed approach delivers actionable, fine-grained insights that lay both theoretical and practical foundations for building more robust and interpretable evaluation systems.

This study addresses the lack of objective validation criteria in existing threat modeling approaches, which often rely on expert judgment and are thus prone to omissions or inconsistencies. To overcome this limitation, the authors propose a quantifiable and reproducible evaluation methodology based on benchmark applications with known vulnerabilities—specifically AzureGoat and VulnBank. Using only architectural diagrams, data flow diagrams, and their textual descriptions as input, the approach evaluates the vulnerability coverage of ThreMoLIA, an LLM-assisted threat modeling system, against Microsoft Threat Modeling Tool. Experimental results demonstrate that ThreMoLIA achieves consistently higher vulnerability coverage across both benchmark applications. This work represents the first effort to employ real-world vulnerable applications as a validation benchmark for threat modeling, effectively mitigating the shortcomings inherent in traditional expert-based assessments.