data lifecycle management

Practices and systems for persisting, versioning, and governing knowledge and telemetry (including semantic metadata) to detect stale or conflicting information and maintain integrity across time-series joins and long‑lived memory stores.

datalifecyclemanagement

12-Month Skill Trend

Momentum and market value over time
Trending
Score
+20 in 12 mo
96
12 mo agoNow
Career
Value
+$12K in 12 mo
$42K/year
12 mo agoNow

Recommended Survey Paper

Quick overview of the field
View more

Must-Read Papers

Most classic and influential ideas
View more

This work addresses the challenge of knowledge evolution in traditional retrieval-augmented generation (RAG), which often retrieves outdated facts. The authors propose MemStrata, a temporally aware retrieval memory system that introduces, for the first time, a dual-temporal ledger mechanism. This mechanism automatically deprecates superseded facts by applying deterministic replacement rules to (subject, predicate, object) triples, eliminating the need for similarity thresholds or large language model intervention. MemStrata maintains strong static knowledge recall while significantly improving temporal accuracy. Experimental results show that on six local benchmarks, MemStrata matches RAG in static performance, achieves 0.95–1.00 accuracy on evolving knowledge (compared to RAG’s 0.20–0.47), reduces outdated errors to nearly 0%, and incurs only 2.1 seconds of retrieval latency.

Evolving KnowledgeFact SupersessionRetrieval-Augmented Generation

In the era of large language models, traditional record-centric data engineering struggles to meet the demand for organizational knowledge as executable infrastructure. This work proposes a novel paradigm—knowledge architecture—that systematically reimagines core data engineering mechanisms by upgrading ETL, data lineage, and catalogs into knowledge ingestion, change detection, provenance, and knowledge catalogs. It introduces knowledge views and a three-tier layered model (raw–refined–operational) to structure knowledge effectively. By integrating emerging standards such as LLM Wiki and Open Knowledge Format (OKF), this study formally defines knowledge architecture for the first time and establishes a theoretical framework that supports knowledge representation, governance, and operational delivery, enabling direct invocation of organizational knowledge by humans, agents, workflows, and models alike.

enterprise AI systemsknowledge architectureknowledge artifacts

The Past Still Matters: A Temporally-Valid Data Discovery System

Oct 15, 2025
ME
Mahdi Esmailoghli
🏛️ Humboldt-Universität zu Berlin

Existing data discovery methods largely overlook the temporal dynamics of data relationships, particularly hindering timeliness-aware retrieval when explicit temporal metadata is absent. To address this, we propose “temporal-effective data discovery”—a novel paradigm introducing the first time-integrated data discovery system. Its core comprises four techniques: version discovery, temporal lineage inference, change-log synthesis, and time-aware querying. Crucially, the system models semantic evolution of data over time without relying on original timestamps, enabling dynamic lineage tracing and version-level precise retrieval. Experimental evaluation on enterprise and public data lakes demonstrates significant improvements in accuracy and practicality for timeliness-sensitive queries. Our work establishes a new research direction in data discovery centered on temporal evolution, advancing both theoretical foundations and real-world applicability.

Addressing time-varying relevance in data discoveryDeveloping techniques for time-aware data lake interactionIncorporating temporal dimension without explicit metadata

Use of a Structured Knowledge Base Enhances Metadata Curation by Large Language Models

Apr 08, 2024
SS
Sowmya S. Sundaram
🏛️ Stanford University

This study addresses the low accuracy of large language models (LLMs) in FAIR-compliance validation of biosample metadata. We propose a structured-knowledge-guided prompting method, integrating the CEDAR template repository, domain-specific data dictionaries, and GPT-4 to construct a metadata standards-conformance verification framework—demonstrated on human lung cancer biosamples. Experimental results show that incorporating structured knowledge significantly improves field-level standards compliance from 79% to 97% (p < 0.01), providing the first empirical evidence that structured knowledge bases can overcome performance bottlenecks inherent to purely text-based LLM prompting in metadata governance. Our approach establishes a novel paradigm for automated, high-accuracy, and interpretable FAIR metadata quality control, enabling scalable, standards-aware curation of biomedical metadata.

Enhance metadata standards adherenceImprove metadata curation automationIntegrate structured knowledge with LLMs

Semantic Web: Past, Present, and Future

Dec 22, 2024
AS
A. Scherp
🏛️ Ulm University | Carl Zeiss SMT GmbH | Charles University | TU Wien | Leibniz University of Hannover | TIB-Leibniz Information Centre for Science and Technology

This paper addresses two critical gaps in the evolution of the Semantic Web: (1) the theoretical lag behind practical applications, and (2) insufficient integration of trustworthiness mechanisms with AI. To bridge these gaps, we propose a unified analytical framework that synergistically integrates classical semantic technologies with modern AI. Methodologically, we extend the canonical “layered cake” model into a novel three-dimensional paradigm encompassing trustworthy computing, industrial validation, and LLM–KG co-adaptation—systematically unifying RDF/OWL representation, rule-based reasoning, distributed SPARQL query processing, knowledge graph embedding, graph neural networks, and LLM–KG alignment techniques. Our contributions include: (1) a comprehensive technology landscape charting 50 years of Semantic Web development; (2) a clarified integration roadmap for knowledge graphs and AI—particularly large language models; and (3) theoretical foundations and practical guidelines for building next-generation semantic infrastructure that is trustworthy, interpretable, and adaptive.

Enhance traditional Semantic Web concepts with recent developments like provenanceExplore machine learning methods on knowledge graphs and language model relationsRecap classical foundations and modern applications of Semantic Web technologies

Latest Papers

What's happening recently
View more

Current AI systems rely heavily on manual auditing and documentation, which hinders scalable governance for automated services. This work proposes Ontological Knowledge Blocks (OKBs), a novel framework that formalizes regulatory obligations as quintuples comprising ontologies, SHACL rules, evidence requirements, and provenance links. By leveraging RDF/OWL modeling, PROV-O for provenance tracking, and an intermediate representation–driven deterministic compiler, the approach enables dynamic switching of governance configurations without modifying service code. Evaluation in an AI-assisted HPC scheduling scenario demonstrates that compliance checks are configuration-sensitive, violations accumulate strictly additively, SHACL validation incurs only 12.6–100.3 milliseconds of latency, and the Combined configuration provides the most comprehensive coverage.

AI governanceautomated verificationcompliance

Existing RDF systems lack standardized message boundaries in event-driven and streaming contexts, resulting in poor interoperability and inefficient processing. This work proposes the concept of “RDF Messages,” formally defining, for the first time, a message structure with RDF datasets as atomic units and explicitly delineating message boundaries to support streaming, storage, and reproducible replay. Built upon W3C Semantic Web standards, the approach introduces a unified encapsulation mechanism that operates across serialization and transport layers, complemented by configuration profiles that describe message structure and processing policies. The proposed framework effectively enables incremental consumption in scenarios such as IoT observations and SPARQL CONSTRUCT results, laying the foundation for RDF message streams and logs while advancing related W3C standardization efforts.

event-driven systemsinteroperabilitymessage boundaries

Current large language model agents lack a persistent memory mechanism that is maintainable, updatable, and capable of providing evidence across sessions. This work proposes a topic-document-based textual persistent memory architecture that organizes memories into structured thematic units, temporarily buffers new observations, and periodically integrates them to support factual revision while preserving metadata. Additionally, it introduces an LLM-driven iterative tool-calling mechanism for precise retrieval. The approach significantly enhances the coherence, updatability, and retrieval effectiveness of long-term memory, achieving a 64.7% overall score on MemoryAgentBench. Ablation studies confirm the complementary benefits of maintaining thematic structure and performing iterative evidence verification.

evidence aggregationfact revisionLLM agent

This study addresses the fragmentation of research software and its associated scholarly resources—such as publications and datasets—across disparate platforms, which hinders reproducibility and cross-domain analysis due to a lack of unified semantic links. To bridge this gap, the authors construct a large-scale RDF knowledge graph comprising 81 million triples, integrating approximately 200,000 GitHub repositories with external academic knowledge graphs including SemOpenAlex, LPWC, and MLSea-KG. This integration enables unified semantic modeling of software alongside scholarly entities such as authors, papers, and datasets. The resulting knowledge graph supports cross-platform provenance tracing and complex semantic queries, significantly enhancing the capacity to assess software reproducibility and analyze its long-term sustainability within the scientific ecosystem.

knowledge graphreproducibilityresearch software

Databases continuously evolve through operations such as schema changes, version updates, and data transformations; however, existing approaches typically address these functionalities in isolation, lacking a unified abstraction. This work proposes the first integrated model that unifies continuous schema evolution, version management, and data transformation within a single framework. Built upon general-purpose computational primitives, the model supports operation provenance, conditional update propagation, and change alerts, while employing a declarative mechanism to manage the co-evolution of dependent artifacts—including views and machine learning models. A prototype system implements this framework using an enhanced, parameterized Prolly Tree—a Merkle tree–inspired data structure—to construct a relational-like engine. Experimental evaluation demonstrates that the proposed approach is both feasible and offers tunable performance across diverse evolution scenarios.

continuous data evolutiondata versioningdatabase transformation

Hot Scholars

RL

Rodrigo Laigner

University of Copenhagen
Data-intensive ApplicationsCloud Data ManagementEvent-based Systems
YZ

Yongluan Zhou

Professor of Computer Science, University of Copenhagen
Database SystemsEvent-Based SystemsDistributed Systems
SD

Stefan Decker

Fraunhofer FIT, RWTH Aachen University
Computer ScienceSemantic WebLinked DataKnowledge Graphs
AL

Antonella Longo

Associate Professor, University of Salento
Big Data ManagementCyber Physical Social SystemsSmart Cities and citizen scienceService com