Plug it and Play on Logs: A Configuration-Free Statistic-Based Log Parser

2025-08-12

AI Summary

Existing statistical log parsers suffer from limited accuracy and poor cross-system generalizability due to reliance on the constant-position assumption and manual hyperparameter tuning, rendering them inferior to semantic approaches based on pretrained language models. This paper proposes PIPLUP—a configuration-free, purely statistical, and universal log parser. Methodologically, PIPLUP abandons positional priors and employs data-agnostic parameters; it integrates dynamic token classification with context-aware clustering, eliminating dependence on pretrained models or external resources. Evaluated on multiple open-source benchmarks, PIPLUP achieves higher parsing accuracy than mainstream statistical parsers (e.g., Drain) and matches the performance of the state-of-the-art unsupervised semantic parser LUNAR. Furthermore, it supports zero-configuration CPU deployment, exhibits low latency, and demonstrates strong practical utility.

Technology Category

Natural Language Processing: Interpretability, Analysis, and Evaluation of NLP ModelsMachine Learning: Statistical Relational/Logic LearningPlanning, Routing, and Scheduling: Planning with Language Models

Application Category

Search and Retrieval-Augmented AI: Multilingual and cross-lingual Web searchSemantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsEconomics, Online Markets and Human Computation: Cost models of using LLMs in production systems

Abstract

Log parsing is an essential task in log analysis, and many tools have been designed to accomplish it. Existing log parsers can be categorized into statistic-based and semantic-based approaches. In comparison to semantic-based parsers, existing statistic-based parsers tend to be more efficient, require lower computational costs, and be more privacy-preserving thanks to on-premise deployment, but often fall short in their accuracy (e.g., grouping or parsing accuracy) and generalizability. Therefore, it became a common belief that statistic-based parsers cannot be as effective as semantic-based parsers since the latter could take advantage of external knowledge supported by pretrained language models. Our work, however, challenges this belief with a novel statistic-based parser, PIPLUP. PIPLUP eliminates the pre-assumption of the position of constant tokens for log grouping and relies on data-insensitive parameters to overcome the generalizability challenge, allowing "plug and play" on given log files. According to our experiments on an open-sourced large log dataset, PIPLUP shows promising accuracy and generalizability with the data-insensitive default parameter set. PIPLUP not only outperforms the state-of-the-art statistic-based log parsers, Drain and its variants, but also obtains a competitive performance compared to the best unsupervised semantic-based log parser (i.e., LUNAR). Further, PIPLUP exhibits low time consumption without GPU acceleration and external API usage; our simple, efficient, and effective approach makes it more practical in real-world adoptions, especially when costs and privacy are of major concerns.

Problem

• Improves accuracy of statistic-based log parsers
• Enhances generalizability without data-sensitive parameters
• Ensures efficiency and privacy in log parsing

Innovation

• Configuration-free statistic-based log parser
• Data-insensitive parameters for generalizability
• Efficient without GPU or external APIs