Data Cleaning of Data Streams

2025-07-28

AI Summary

Traditional static data cleaning methods exhibit inconsistency and poor adaptability when applied to dynamic data streams—such as continuous temporal sensor measurements (e.g., temperature, illumination)—posing significant challenges for real-time error detection and correction. Method: This work identifies the inherent non-stationarity of stream cleaning and establishes the first theoretical framework specifically designed for streaming environments, formally characterizing its unique challenges: timeliness constraints, state evolution, and incremental validation. Based on this framework, we design a prototype system and conduct empirical evaluations across diverse streaming scenarios. Contribution/Results: Our analysis reveals substantial performance volatility in existing approaches, pinpointing three critical factors: windowing strategy, error propagation patterns, and data arrival rate. The framework provides a scalable theoretical foundation and a rigorous evaluation methodology for stream data cleaning, enabling principled design and comparative assessment of future techniques.

Technology Category

• Data Mining & Knowledge Management: Data Stream Mining
• Machine Learning: Time-Series/Data Streams
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Web Mining and Content Analysis: Web data integration and cleaning
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Streaming data can arise from a variety of contexts. Important use cases are continuous sensor measurements such as temperature, light or radiation values. In the process, streaming data may also contain data errors that should be cleaned before further use. Many studies from science and practice focus on data cleaning in a static context. However, in terms of data cleaning, streaming data has particularities that distinguish it from static data. In this paper, we have therefore undertaken an intensive exploration of data cleaning of data streams. We provide a detailed analysis of the applicability of data cleaning to data streams. Our theoretical considerations are evaluated in comprehensive experiments. Using a prototype framework, we show that cleaning is not consistent when working with data streams. An additional contribution is the investigation of requirements for streaming technologies in context of data cleaning.

Problem

• Addressing data cleaning challenges in streaming data
• Analyzing applicability of cleaning methods for data streams
• Investigating requirements for streaming technologies in cleaning

Innovation

• Explores data cleaning for streaming data
• Evaluates cleaning methods via prototype framework
• Investigates requirements for streaming technologies