🤖 AI Summary
In cross-library data science workflows, existing data lineage representations are tightly coupled to library-specific data models and operational paradigms, hindering tasks such as cross-library debugging. This paper proposes XProv, a unified intermediate representation architecture that integrates concrete data transformation graphs with abstract logical schemas to enable parameterized lineage modeling and correlation analysis for both known and unknown cross-library operations. Inspired by compiler intermediate representations, XProv decouples lineage constraints into extensible logical schemas and supports automatic schema inference from execution traces. Experiments demonstrate that XProv is the first approach to achieve unified logical lineage representation and schema learning across multiple libraries, providing a scalable foundational framework for cross-library provenance tracking, debugging, and verification.
📝 Abstract
Data science workflows often integrate functionalities from a diverse set of libraries and frameworks. Tasks such as debugging require data lineage that crosses library boundaries. The problem is that the way that "lineage" is represented is often intimately tied to particular data models and data manipulation paradigms. Inspired by the use of intermediate representations (IRs) in cross-library performance optimizations, this vision paper proposes a similar architecture for lineage - how do we specify logical lineage across libraries in a common parameterized way? In practice, cross-library workflows will contain both known operations and unknown operations, so a key design of XProv to link both materialized lineage graphs of data transformations and the aforementioned abstracted logical patterns. We further discuss early ideas on how to infer logical patterns when only the materialized graphs are available.