A Polynomial Space Lower Bound for Diameter Estimation in Dynamic Streams

2025-10-06

AI Summary

This paper studies the problem of approximating the diameter (i.e., the maximum pairwise distance) of a dynamic point set in a metric space $M$, where the point set is specified by a frequency vector undergoing insertions and deletions in the dynamic streaming model. The goal is to estimate the diameter up to a constant multiplicative factor using minimal space. Methodologically, the work establishes a tight connection between dynamic streaming algorithms and scale-invariant linear sketches, leveraging graph-theoretic minrank properties and communication complexity lower bounds. It proves that any $c$-approximation algorithm requires $Omega(n^{1/c})$ space, and provides a matching $O(n^{1/c})$ upper bound via an explicit construction. This resolves the space complexity of diameter estimation in dynamic streams: unlike the insertion-only model, constant-factor approximation necessarily incurs polynomial space dependence—revealing a fundamental separation between the two models.

Technology Category

• Data Mining & Knowledge Management: Data Stream Mining
• Search and Optimization: Distributed Search
• Knowledge Representation and Reasoning: Computational Complexity of Reasoning

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Security and Privacy: Large-scale security measurements
• Web Mining and Content Analysis: Models for Web evolution

Abstract

We study the space complexity of estimating the diameter of a subset of points in an arbitrary metric space in the dynamic (turnstile) streaming model. The input is given as a stream of updates to a frequency vector $x in mathbb{Z}_{geq 0}^n$, where the support of $x$ defines a multiset of points in a fixed metric space $M = ([n], mathsf{d})$. The goal is to estimate the diameter of this multiset, defined as $max{mathsf{d}(i,j) : x_i, x_j > 0}$, to a specified approximation factor while using as little space as possible. In insertion-only streams, a simple $O(log n)$-space algorithm achieves a 2-approximation. In sharp contrast to this, we show that in the dynamic streaming model, any algorithm achieving a constant-factor approximation to diameter requires polynomial space. Specifically, we prove that a $c$-approximation to the diameter requires $n^{Ω(1/c)}$ space. Our lower bound relies on two conceptual contributions: (1) a new connection between dynamic streaming algorithms and linear sketches for {em scale-invariant} functions, a class that includes diameter estimation, and (2) a connection between linear sketches for diameter and the {em minrank} of graphs, a notion previously studied in index coding. We complement our lower bound with a nearly matching upper bound, which gives a $c$-approximation to the diameter in general metrics using $n^{O(1/c)}$ space.

Problem

• Estimate diameter of point subsets in dynamic streaming model
• Require polynomial space for constant-factor diameter approximation
• Establish space complexity lower bound for diameter estimation

Innovation

• Dynamic streaming model for diameter estimation
• Lower bound requires polynomial space complexity
• Connection between linear sketches and minrank