Towards Small Language Models for Security Query Generation in SOC Workflows

2025-12-07

AI Summary

To address the query efficiency bottleneck arising from SOC analysts’ reliance on expert-level KQL proficiency, this paper proposes a lightweight, security-domain-specific natural language-to-KQL (NL2KQL) translation method. Methodologically, it introduces a two-stage framework comprising error-aware prompting, LoRA fine-tuning guided by reasoning distillation, and a small-model generation–large-model discrimination paradigm, augmented with lightweight retrieval enhancement. The key contribution lies in jointly optimizing security-specific semantic understanding and computational cost control. Evaluated on Microsoft’s NL2KQL and Sentinel datasets, the approach achieves syntax/semantic accuracy of 98.7%/90.6% and 96.4%/83.1%, respectively, while reducing token consumption by an order of magnitude. This significantly enhances the generalizability and deployment cost-efficiency of small language models (SLMs) in real-world security operations.

Technology Category

• Natural Language Processing: Safety and Robustness
• Data Mining & Knowledge Management: Conversational Systems for Recommendation & Retrieval
• Search and Optimization: Learning to Search

Application Category

• Search and Retrieval-Augmented AI: Large language models for search
• Semantics and Knowledge: Methods to enhance, augment, integrate or synergize semantic models such as knowledge graphs and LLMs
• Security and Privacy: Large-scale security measurements

Abstract

Analysts in Security Operations Centers routinely query massive telemetry streams using Kusto Query Language (KQL). Writing correct KQL requires specialized expertise, and this dependency creates a bottleneck as security teams scale. This paper investigates whether Small Language Models (SLMs) can enable accurate, cost-effective natural-language-to-KQL translation for enterprise security. We propose a three-knob framework targeting prompting, fine-tuning, and architecture design. First, we adapt existing NL2KQL framework for SLMs with lightweight retrieval and introduce error-aware prompting that addresses common parser failures without increasing token count. Second, we apply LoRA fine-tuning with rationale distillation, augmenting each NLQ-KQL pair with a brief chain-of-thought explanation to transfer reasoning from a teacher model while keeping the SLM compact. Third, we propose a two-stage architecture that uses an SLM for candidate generation and a low-cost LLM judge for schema-aware refinement and selection. We evaluate nine models (five SLMs and four LLMs) across syntax correctness, semantic accuracy, table selection, and filter precision, alongside latency and token cost. On Microsoft's NL2KQL Defender Evaluation dataset, our two-stage approach achieves 0.987 syntax and 0.906 semantic accuracy. We further demonstrate generalizability on Microsoft Sentinel data, reaching 0.964 syntax and 0.831 semantic accuracy. These results come at up to 10x lower token cost than GPT-5, establishing SLMs as a practical, scalable foundation for natural-language querying in security operations.

Problem

• Enabling accurate natural-language-to-KQL translation for security analysts
• Reducing dependency on specialized expertise for writing Kusto queries
• Providing a cost-effective and scalable solution for SOC workflows

Innovation

• Error-aware prompting addresses parser failures efficiently
• LoRA fine-tuning with rationale distillation transfers reasoning compactly
• Two-stage architecture uses SLM generation and LLM refinement