🤖 AI Summary
Analyzing massive, heterogeneous, and scale-free system logs remains challenging due to their volume, diversity, and lack of standardized structure. Method: This paper proposes Time Curves, a novel log analysis pipeline integrating log clustering, event detection, LLM-driven summarization, multidimensional scaling (MDS), and Time Curves visualization. It introduces a semi-metric distance function tailored for log events and is the first to apply Time Curves to log analysis—enabling concurrent, overlaid projections for temporal trend identification and fine-grained anomaly detection. Contribution/Results: The method requires no prior knowledge, automatically extracts dominant events from multi-source logs, and achieves joint semantic-temporal modeling with full interpretability. Evaluated on distributed systems, it simultaneously reveals global behavioral patterns and localized anomalies, significantly reducing time-to-diagnosis for faults, performance bottlenecks, and security threats.
📝 Abstract
Logs are crucial for analyzing large-scale software systems, offering insights into system health, performance, security threats, potential bugs, etc. However, their chaotic nature$unicode{x2013}$characterized by sheer volume, lack of standards, and variability$unicode{x2013}$makes manual analysis complex. The use of clustering algorithms can assist by grouping logs into a smaller set of templates, but lose the temporal and relational context in doing so. On the contrary, Large Language Models (LLMs) can provide meaningful explanations but struggle with processing large collections efficiently. Moreover, representation techniques for both approaches are typically limited to either plain text or traditional charting, especially when dealing with large-scale systems. In this paper, we combine clustering and LLM summarization with event detection and Multidimensional Scaling through the use of Time Curves to produce a holistic pipeline that enables efficient and automatic summarization of vast collections of software system logs. The core of our approach is the proposal of a semimetric distance that effectively measures similarity between events, thus enabling a meaningful representation. We show that our method can explain the main events of logs collected from different applications without prior knowledge. We also show how the approach can be used to detect general trends as well as outliers in parallel and distributed systems by overlapping multiple projections. As a result, we expect a significant reduction of the time required to analyze and resolve system-wide issues, identify performance bottlenecks and security risks, debug applications, etc.