LLM-Assisted AHP for Explainable Cyber Range Evaluation

2025-12-11

AI Summary

Critical infrastructure cyber ranges (CRs) lack standardized, objective, and reproducible evaluation methodologies. Method: This paper proposes the first explainable evaluation framework integrating the Analytic Hierarchy Process (AHP) with large language models (LLMs). It innovatively employs LLMs to emulate multidisciplinary expert panels, enabling automated, traceable dynamic weight generation without human experts. The framework establishes a five-dimensional metric system—encompassing technical fidelity, training efficacy, scalability, usability, and security realism—and produces quantitative scores alongside attribution-aware interpretability. Contribution/Results: Experimental validation demonstrates that the framework significantly outperforms conventional approaches in discriminability, inter-method consistency, and explanatory transparency, enabling rigorous cross-platform CR comparison and precise capability gap identification.

Technology Category

• Machine Learning: Evaluation and Analysis
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

Cyber Ranges (CRs) have emerged as prominent platforms for cybersecurity training and education, especially for Critical Infrastructure (CI) sectors that face rising cyber threats. One way to address these threats is through hands-on exercises that bridge IT and OT domains to improve defensive readiness. However, consistently evaluating whether a CR platform is suitable and effective remains a challenge. This paper proposes an evaluation framework for CRs, emphasizing mission-critical settings by using a multi-criteria decision-making approach. We define a set of evaluation criteria that capture technical fidelity, training and assessment capabilities, scalability, usability, and other relevant factors. To weight and aggregate these criteria, we employ the Analytic Hierarchy Process (AHP), supported by a simulated panel of multidisciplinary experts implemented through a Large Language Model (LLM). This LLM-assisted expert reasoning enables consistent and reproducible pairwise comparisons across criteria without requiring direct expert convening. The framework's output equals quantitative scores that facilitate objective comparison of CR platforms and highlight areas for improvement. Overall, this work lays the foundation for a standardized and explainable evaluation methodology to guide both providers and end-users of CRs.

Problem

• Proposes an evaluation framework for Cyber Ranges using multi-criteria decision-making
• Employs LLM-assisted AHP to weight criteria without direct expert convening
• Provides quantitative scores for objective comparison and improvement of CR platforms

Innovation

• LLM-assisted AHP for expert reasoning
• Multi-criteria framework for cyber range evaluation
• Quantitative scores for objective platform comparison