The Case for External Graph Sketching

2025-04-24

AI Summary

Existing sketching algorithms for graph semi-streaming models require $O(V cdot ext{polylog},V)$ memory—prohibitive for large-scale graphs. To address this, we propose the **external semi-streaming model**, which permits external-memory access via block I/O, drastically reducing main-memory footprint while bounding I/O cost. Our contributions are: (1) the first formal definition of the external semi-streaming model; (2) a generic framework for automatically converting vertex-based sketches into external-memory–aware variants; (3) a tight lower bound on I/O complexity; and (4) the first algorithms achieving both $O(V cdot ext{polylog},V)$ space and near-linear I/O cost for fundamental graph problems—including connectivity, bipartiteness, MST weight, $k$-edge connectivity, minimum cut, cut sparsification, and densest subgraph density—surpassing all prior sketching and external-memory approaches in both space and I/O efficiency.

Technology Category

• Machine Learning: Graph-based Machine Learning
• Data Mining & Knowledge Management: Data Stream Mining
• Search and Optimization: Distributed Search

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Semantics and Knowledge: Methods, algorithms and applications for the development of semantic models, knowledge graphs and other forms of structured data models with machine-interpretable semantics
• Web Mining and Content Analysis: Models for Web evolution

Abstract

Algorithms in the data stream model use $O(polylog(N))$ space to compute some property of an input of size $N$, and many of these algorithms are implemented and used in practice. However, sketching algorithms in the graph semi-streaming model use $O(V polylog(V))$ space for a $V$-vertex graph, and the fact that implementations of these algorithms are not used in the academic literature or in industrial applications may be because this space requirement is too large for RAM on today's hardware. In this paper we introduce the external semi-streaming model, which addresses the aspects of the semi-streaming model that limit its practical impact. In this model, the input is in the form of a stream and $O(V polylog(V))$ space is available, but most of that space is accessible only via block I/O operations as in the external memory model. The goal in the external semi-streaming model is to simultaneously achieve small space and low I/O cost. We present a general transformation from any vertex-based sketch algorithm to one which has a low sketching cost in the new model. We prove that this automatic transformation is tight or nearly (up to a $O(log(V))$ factor) tight via an I/O lower bound for the task of sketching the input stream. Using this transformation and other techniques, we present external semi-streaming algorithms for connectivity, bipartiteness testing, $(1+epsilon)$-approximating MST weight, testing k-edge connectivity, $(1+epsilon)$-approximating the minimum cut of a graph, computing $epsilon$-cut sparsifiers, and approximating the density of the densest subgraph. These algorithms all use $O(V poly(log(V), epsilon^{-1},k)$ space. For many of these problems, our external semi-streaming algorithms outperform the state of the art algorithms in both the sketching and external-memory models.

Problem

• Addresses impractical space usage in graph semi-streaming algorithms
• Introduces external semi-streaming model for low space and I/O cost
• Provides efficient algorithms for graph problems in new model

Innovation

• Introduces external semi-streaming model for graphs
• Transforms vertex-based sketches to low I/O cost
• Achieves efficient space and I/O for graph problems