Multi-Stage Retrieval for Operational Technology Cybersecurity Compliance Using Large Language Models: A Railway Casestudy

2025-04-18

AI Summary

To address low efficiency and expert shortages in operational technology (OT) cybersecurity compliance verification for critical infrastructure (e.g., railways), this paper proposes and validates the Parallel Compliance Architecture (PCA)—the first automated OT cybersecurity standard (OTCS) verification framework integrating context-enhanced reasoning over standard textual specifications, supporting IEC 62443 and IEC 63452. Methodologically, PCA unifies multi-stage semantic retrieval, a compliance knowledge graph, retrieval-augmented generation (RAG), and standardized prompt engineering, leveraging GPT-4o and Claude-3.5-haiku for precise, auditable inference. Its novelty includes a three-dimensional evaluation metric assessing correctness, logical coherence, and hallucination detection. Experiments demonstrate that PCA significantly outperforms baselines in response accuracy and reasoning quality; retrieval augmentation effectively mitigates hallucinations, and overall assessment efficiency improves by 3.2×. PCA establishes a scalable, verifiable, AI-augmented paradigm for OTCS compliance validation.

Technology Category

• Constraint Satisfaction and Optimization: Satisfiability Modulo Theories
• Search and Optimization: Metareasoning and Metaheuristics
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Security and Privacy: Large-scale security measurements

Abstract

Operational Technology Cybersecurity (OTCS) continues to be a dominant challenge for critical infrastructure such as railways. As these systems become increasingly vulnerable to malicious attacks due to digitalization, effective documentation and compliance processes are essential to protect these safety-critical systems. This paper proposes a novel system that leverages Large Language Models (LLMs) and multi-stage retrieval to enhance the compliance verification process against standards like IEC 62443 and the rail-specific IEC 63452. We first evaluate a Baseline Compliance Architecture (BCA) for answering OTCS compliance queries, then develop an extended approach called Parallel Compliance Architecture (PCA) that incorporates additional context from regulatory standards. Through empirical evaluation comparing OpenAI-gpt-4o and Claude-3.5-haiku models in these architectures, we demonstrate that the PCA significantly improves both correctness and reasoning quality in compliance verification. Our research establishes metrics for response correctness, logical reasoning, and hallucination detection, highlighting the strengths and limitations of using LLMs for compliance verification in railway cybersecurity. The results suggest that retrieval-augmented approaches can significantly improve the efficiency and accuracy of compliance assessments, particularly valuable in an industry facing a shortage of cybersecurity expertise.

Problem

• Enhancing compliance verification for railway cybersecurity using LLMs
• Addressing vulnerabilities in OT systems through multi-stage retrieval
• Improving accuracy of standards compliance (IEC 62443/63452) assessments

Innovation

• Leverages Large Language Models for compliance verification
• Uses multi-stage retrieval to enhance accuracy
• Incorporates regulatory standards context via Parallel Architecture