Compact Data Structures for Network Telemetry

2023-11-05

arXiv.org

AI Summary

Conventional network telemetry frameworks struggle to support fine-grained traffic measurement, performance diagnostics, and attack detection under stringent memory and computational constraints of high-speed network devices. Method: This paper proposes a lightweight, real-time online telemetry framework that systematically integrates compact data structures—including Bloom filter variants, Count-Min Sketch, and HyperLogLog—with streaming algorithms, hierarchical sampling, and P4-programmable data-plane co-design to comply with hardware limitations. Contribution/Results: Evaluated at line rate exceeding 100 Gbps, the framework reduces memory footprint by over 60% compared to state-of-the-art approaches while maintaining sub-1% flow frequency estimation error. It achieves an optimal trade-off among accuracy, throughput, and resource overhead, thereby significantly enhancing the feasibility and practicality of telemetry in high-bandwidth environments.

Abstract

Collecting and analyzing of network traffic data (network telemetry) plays a critical role in managing modern networks. Network administrators analyze their traffic to troubleshoot performance and reliability problems, and to detect and block cyberattacks. However, conventional traffic-measurement techniques offer limited visibility into network conditions and rely on offline analysis. Fortunately, network devices -- such as switches and network interface cards -- are increasingly programmable at the packet level, enabling flexible analysis of the traffic in place, as the packets fly by. However, to operate at high speed, these devices have limited memory and computational resources, leading to trade-offs between accuracy and overhead. In response, an exciting research area emerged, bringing ideas from compact data structures and streaming algorithms to bear on important networking telemetry applications and the unique characteristics of high-speed network devices. In this paper, we review the research on compact data structures for network telemetry and discuss promising directions for future research.

Problem

• Compact data structures for traffic analysis
• High-speed network device limitations
• Trade-offs between accuracy and overhead

Innovation

• Compact data structures
• Streaming algorithms
• High-speed network devices