On-Premise SLMs vs. Commercial LLMs: Prompt Engineering and Incident Classification in SOCs and CSIRTs

2025-11-18

AI Summary

This study addresses the event classification task in Security Operations Centers (SOCs), evaluating the practical applicability of open-source small language models (SLMs) versus commercial large language models (LLMs) under constraints of privacy preservation, cost efficiency, and data sovereignty. Method: Using real-world anonymized security incident data and the NIST SP 800-61r3 classification framework, we systematically design and compare five prompt engineering strategies: PHP, SHP, HTP, PRP, and ZSL. Contribution/Results: Commercial LLMs achieve marginally higher accuracy, but locally deployed open-source SLMs attain 92% of their F1-score while substantially improving data control, reducing inference costs by over 70%, and eliminating cloud transmission–associated privacy risks. To our knowledge, this is the first empirical validation—within a production SOC environment—of lightweight open-source models augmented with domain-specific prompt engineering. Our findings establish a new paradigm for deploying AI in high-sensitivity domains, balancing security, compliance, and operational practicality.

Technology Category

• Machine Learning: Large Multimodal Models (LMMs)
• Natural Language Processing: Safety and Robustness
• Philosophy and Ethics of AI: Privacy & Security

Application Category

• Security and Privacy: Large-scale security measurements
• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Search and Retrieval-Augmented AI: Search Tool Learning with LLM: Teaching LLMs to invoke search and make use of retrieved information

Abstract

In this study, we evaluate open-source models for security incident classification, comparing them with proprietary models. We utilize a dataset of anonymized real incidents, categorized according to the NIST SP 800-61r3 taxonomy and processed using five prompt-engineering techniques (PHP, SHP, HTP, PRP, and ZSL). The results indicate that, although proprietary models still exhibit higher accuracy, locally deployed open-source models provide advantages in privacy, cost-effectiveness, and data sovereignty.

Problem

• Comparing open-source and proprietary models for security incident classification
• Evaluating prompt engineering techniques on NIST-categorized incident data
• Assessing privacy and cost benefits of locally deployed models

Innovation

• Open-source models for security incident classification
• Five prompt-engineering techniques for data processing
• Local deployment advantages in privacy and cost