An Improved Linear Extractable Sketch Data Structure for Flow Count Statistics

2026-07-22

AI Summary

This work addresses a key limitation in traditional linear sketching methods for streaming data statistics, which rely on the strong assumption that each hash bucket contains only a single key—thereby constraining space efficiency. To overcome this, the authors propose a novel approach that stores randomized linear combinations of multiple keys within each bucket and reconstructs key-value pairs during recovery by solving a sparse linear system. This design effectively relaxes the single-key-per-bucket constraint, achieving substantially improved space efficiency with only a modest increase in computational overhead. Experimental results demonstrate that the proposed method significantly reduces memory consumption while markedly enhancing space utilization for streaming data statistics.

Abstract

Sketch data structures are very useful for computing statistics on streaming data, including network traffic, server requests, and financial transactions. In recent work, FermatSketch was introduced as an underlying data structure used to monitor changes in network states. It is a linear data structure that maintains an associated array of counters and supports listing all key-counter pairs while using almost linear space. Because it is linear, it can be used to monitor changes between two streams with space proportional to the number of items that change. The data structure is based on a hash table, and all key-counter pairs can be successfully listed when there are slots in the table with exactly one key hashed to them. We show how to relax this requirement by using additional computational resources when listing the key-counter pairs, thereby improving space efficiency with only a small overhead when collecting statistics. We achieve this by storing, for each bucket, multiple linear combinations of the counters whose coefficients are generated from the keys. With this information, certain linear systems can be solved to obtain the key-counter pairs. A preliminary experiment shows a significant reduction of memory needed for the data structure. Our work can be viewed as a trade-off between space and time.

Problem

• sketch data structure
• flow count statistics
• space efficiency
• linear extractable sketch
• streaming data

Innovation

• linear sketch
• flow counting
• space-time tradeoff
• hash-based data structure
• streaming algorithms