$K^4$: Online Log Anomaly Detection Via Unsupervised Typicality Learning

2025-07-26

AI Summary

To address critical challenges in log-based anomaly detection—including reliance on fragile log parsing, slow inference, and unrealistic evaluation protocols—this paper proposes an unsupervised, parser-free online detection framework. Our method introduces, for the first time, a lightweight, interpretable four-dimensional typicality descriptor (Precision/Recall/Density/Coverage) based on k-nearest-neighbor statistics, directly mapping log embeddings to anomaly scores without training or fine-tuning. It eliminates log parsing entirely and supports real-time streaming inference. Under a realistic online evaluation protocol—closer to production deployment—our approach achieves AUROC scores of 0.995–0.999, substantially outperforming state-of-the-art methods. Training completes in under 4 seconds, and per-log inference takes only 4 microseconds, representing speedups of several orders of magnitude. The framework is plug-and-play, requiring no model retraining or domain-specific adaptation.

Technology Category

• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Machine Learning: Calibration & Uncertainty Quantification
• Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP Models

Application Category

• Search and Retrieval-Augmented AI: Web evaluation methodologies and metrics
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Web Mining and Content Analysis: Web traffic and log analysis

Abstract

Existing Log Anomaly Detection (LogAD) methods are often slow, dependent on error-prone parsing, and use unrealistic evaluation protocols. We introduce $K^4$, an unsupervised and parser-independent framework for high-performance online detection. $K^4$ transforms arbitrary log embeddings into compact four-dimensional descriptors (Precision, Recall, Density, Coverage) using efficient k-nearest neighbor (k-NN) statistics. These descriptors enable lightweight detectors to accurately score anomalies without retraining. Using a more realistic online evaluation protocol, $K^4$ sets a new state-of-the-art (AUROC: 0.995-0.999), outperforming baselines by large margins while being orders of magnitude faster, with training under 4 seconds and inference as low as 4 $μ$s.

Problem

• Unsupervised online log anomaly detection
• Parser-independent log analysis framework
• Fast and accurate anomaly scoring

Innovation

• Unsupervised parser-independent log anomaly detection
• Four-dimensional descriptors via k-NN statistics
• Lightweight fast training and inference