The Past Still Matters: A Temporally-Valid Data Discovery System

2025-10-15

AI Summary

Existing data discovery methods largely overlook the temporal dynamics of data relationships, particularly hindering timeliness-aware retrieval when explicit temporal metadata is absent. To address this, we propose “temporal-effective data discovery”—a novel paradigm introducing the first time-integrated data discovery system. Its core comprises four techniques: version discovery, temporal lineage inference, change-log synthesis, and time-aware querying. Crucially, the system models semantic evolution of data over time without relying on original timestamps, enabling dynamic lineage tracing and version-level precise retrieval. Experimental evaluation on enterprise and public data lakes demonstrates significant improvements in accuracy and practicality for timeliness-sensitive queries. Our work establishes a new research direction in data discovery centered on temporal evolution, advancing both theoretical foundations and real-world applicability.

Technology Category

• Data Mining & Knowledge Management: Mining of Spatial, Temporal or Spatio-Temporal Data
• Machine Learning: Time-Series/Data Streams
• Knowledge Representation and Reasoning: Geometric, Spatial, and Temporal Reasoning

Application Category

• Web Mining and Content Analysis: Topic discovery and tracking
• Graph Algorithms and Modeling for the Web: Querying, indexing, and retrieval in Web-related graphs
• Security and Privacy: Data transparency and provenance

Abstract

Over the past decade, the proliferation of public and enterprise data lakes has fueled intensive research into data discovery, aiming to identify the most relevant data from vast and complex corpora to support diverse user tasks. Significant progress has been made through the development of innovative index structures, similarity measures, and querying infrastructures. Despite these advances, a critical aspect remains overlooked: relevance is time-varying. Existing discovery methods largely ignore this temporal dimension, especially when explicit date/time metadata is missing. To fill this gap, we outline a vision for a data discovery system that incorporates the temporal dimension of data. Specifically, we define the problem of temporally-valid data discovery and argue that addressing it requires techniques for version discovery, temporal lineage inference, change log synthesis, and time-aware data discovery. We then present a system architecture to deliver these techniques, before we summarize research challenges and opportunities. As such, we lay the foundation for a new class of data discovery systems, transforming how we interact with evolving data lakes.

Problem

• Addressing time-varying relevance in data discovery
• Incorporating temporal dimension without explicit metadata
• Developing techniques for time-aware data lake interaction

Innovation

• Incorporates temporal dimension into data discovery
• Uses version discovery and temporal lineage inference
• Implements time-aware architecture for evolving data lakes