Token Interdependency Parsing (Tipping) - Fast and Accurate Log Parsing

2024-08-01

arXiv.org

AI Summary

To address the challenge of automatic template-parameter separation in massive, complex log data, this paper proposes Tipping—a highly efficient and accurate log parsing method. Methodologically, Tipping introduces (1) a novel token interdependence graph model, leveraging strongly connected component (SCC) analysis for robust template identification, and (2) a near-linear, multi-core parallel architecture enabling sub-second throughput. Evaluated on standard benchmarks, Tipping processes 11 million log lines in just 20 seconds on a laptop—outperforming state-of-the-art methods in both accuracy and speed. Moreover, it significantly enhances downstream anomaly detection performance.

Technology Category

• Natural Language Processing: Syntax — Tagging, Chunking & Parsing
• Machine Learning: Time-Series/Data Streams
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Web Mining and Content Analysis: Web traffic and log analysis
• Security and Privacy: Large-scale security measurements

Abstract

In the last decade, an impressive increase in software adaptions has led to a surge in log data production, making manual log analysis impractical and establishing the necessity for automated methods. Conversely, most automated analysis tools include a component designed to separate log templates from their parameters, commonly referred to as a"log parser". This paper aims to introduce a new fast and accurate log parser, named"Tipping". Tipping combines rule-based tokenizers, interdependency token graphs, strongly connected components, and various techniques to ensure rapid, scalable, and precise log parsing. Furthermore, Tipping is parallelized and capable of running on multiple processing cores with close to linear efficiency. We evaluated Tipping against other state-of-the-art log parsers in terms of accuracy, performance, and the downstream task of anomaly detection. Accordingly, we found that Tipping outperformed existing methods in accuracy and performance in our evaluations. More in-depth, Tipping can parse 11 million lines of logs in less than 20 seconds on a laptop machine. Furthermore, we re-implemented a parallelized version of the past IpLom algorithm to demonstrate the effect of parallel processing, and it became the second-fastest parser. As logs keep growing in volume and complexity, the software engineering community needs to ensure automated log analysis tools keep up with the demand, being capable of efficiently handling massive volumes of logs with high accuracy. Tipping's robustness, versatility, efficiency, and scalability make it a viable tool for the modern automated log analysis task.

Problem

• Automating log template separation from parameters
• Handling massive volumes of logs efficiently and accurately
• Improving performance and scalability for log parsing

Innovation

• Combines rule-based tokenizers and interdependency token graphs
• Uses strongly connected components for precise log parsing
• Parallelized processing with near-linear efficiency scaling