Enabling Secure and Ephemeral AI Workloads in Data Mesh Environments

2025-05-31

AI Summary

In large enterprises operating data meshes, AI and data teams struggle to rapidly provision, secure, and dynamically scale experimental workloads within highly governed, multi-cloud/hybrid-cloud ICT environments. This paper proposes a zero-state, policy-driven Kubernetes cluster provisioning method leveraging immutable container OSes and infrastructure-as-code (IaC), enabling vendor-agnostic, reproducible, self-service infrastructure with second-scale cluster lifecycle management. By eliminating node state persistence, the approach significantly reduces operational complexity and cost, offering greater portability, lightweightness, and governance than commercial PaaS solutions. Evaluation demonstrates that, while maintaining security and regulatory compliance, the method accelerates data product iteration by 3–5× and improves interoperability and governance consistency between modern analytics tools and legacy systems.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Scalability of ML Systems
• Humans and AI: Crowd Sourcing and Human Computation

Application Category

• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Semantics and Knowledge: Provenance, trust, security and privacy, and ethical issues in managing semantic data

Abstract

Many large enterprises that operate highly governed and complex ICT environments have no efficient and effective way to support their Data and AI teams in rapidly spinning up and tearing down self-service data and compute infrastructure, to experiment with new data analytic tools, and deploy data products into operational use. This paper proposes a key piece of the solution to the overall problem, in the form of an on-demand self-service data-platform infrastructure to empower de-centralised data teams to build data products on top of centralised templates, policies and governance. The core innovation is an efficient method to leverage immutable container operating systems and infrastructure-as-code methodologies for creating, from scratch, vendor-neutral and short-lived Kubernetes clusters on-premises and in any cloud environment. Our proposed approach can serve as a repeatable, portable and cost-efficient alternative or complement to commercial Platform-as-a-Service (PaaS) offerings, and this is particularly important in supporting interoperability in complex data mesh environments with a mix of modern and legacy compute infrastructure.

Problem

• Enabling secure ephemeral AI workloads in data mesh environments
• Providing on-demand self-service infrastructure for decentralized data teams
• Creating vendor-neutral short-lived Kubernetes clusters efficiently

Innovation

• On-demand self-service data-platform infrastructure
• Immutable container OS and infrastructure-as-code
• Vendor-neutral short-lived Kubernetes clusters