eACGM: Non-instrumented Performance Tracing and Anomaly Detection towards Machine Learning Systems

2025-05-25

AI Summary

To address the challenges of intrusive instrumentation and fragmented cross-layer analysis in full-stack performance monitoring of AI/ML systems, this paper proposes an eBPF-based non-intrusive monitoring framework. The framework integrates eBPF kernel-level tracing with libnvml-driven process-level GPU telemetry to enable zero-modification, real-time collection of performance metrics across GPU hardware, networking, CUDA runtime, Python execution, and PyTorch framework layers. It introduces a novel multi-source heterogeneous time-series modeling approach, coupled with Gaussian Mixture Model (GMM)-driven unsupervised multidimensional anomaly clustering, enabling automated detection and root-cause localization of complex failures—including latency spikes, hardware faults, and communication inefficiencies. Evaluated in multi-node distributed training environments, the framework incurs <3% runtime overhead while achieving high detection accuracy and demonstrating production-grade scalability.

Technology Category

• Machine Learning: Hardware-aware ML
• Data Mining & Knowledge Management: Anomaly/Outlier Detection
• Computer Vision: Multi-modal Vision

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs
• Security and Privacy: Security and privacy of machine learning and AI applications

Abstract

We present eACGM, a full-stack AI/ML system monitoring framework based on eBPF. eACGM collects real-time performance data from key hardware components, including the GPU and network communication layer, as well as from key software stacks such as CUDA, Python, and PyTorch, all without requiring any code instrumentation or modifications. Additionally, it leverages libnvml to gather process-level GPU resource usage information. By applying a Gaussian Mixture Model (GMM) to the collected multidimensional performance metrics for statistical modeling and clustering analysis, eACGM effectively identifies complex failure modes, such as latency anomalies, hardware failures, and communication inefficiencies, enabling rapid diagnosis of system bottlenecks and abnormal behaviors. To evaluate eACGM's effectiveness and practicality, we conducted extensive empirical studies and case analyses in multi-node distributed training scenarios. The results demonstrate that eACGM, while maintaining a non-intrusive and low-overhead profile, successfully captures critical performance anomalies during model training and inference. Its stable anomaly detection performance and comprehensive monitoring capabilities validate its applicability and scalability in real-world production environments, providing strong support for performance optimization and fault diagnosis in large-scale AI/ML systems.

Problem

• Monitors AI/ML systems without code instrumentation
• Detects hardware and software performance anomalies
• Identifies system bottlenecks in distributed training

Innovation

• eBPF-based full-stack AI/ML monitoring framework
• Non-intrusive real-time GPU and software stack tracing
• GMM for anomaly detection in performance metrics