LakeVilla: Multi-Table Transactions for Lakehouses

2025-04-29

AI Summary

Open table formats (OTFs) such as Apache Iceberg and Delta Lake—widely adopted in lakehouse architectures—lack native support for cross-table ACID transactions and crash recovery, leading to non-repeatable reads, stale data, and high operational overhead. This paper presents the first non-intrusive solution that enables multi-table transactions, strong consistency isolation—including for long-running transactions—and fault tolerance on OTFs without modifying their storage formats or computation engines. Our approach introduces log-coordinated lightweight two-phase commit, snapshot version-chain management, atomic metadata writes, and incremental rollback. Evaluation shows minimal performance overhead: only 2.0% on YCSB write workloads and 2.5% on TPC-DS read workloads. The system supports high-concurrency mixed workloads while eliminating dirty reads and non-repeatable reads entirely, thereby ensuring production-grade consistency for cloud-scale analytical data.

Technology Category

• Search and Optimization: Distributed Search
• Constraint Satisfaction and Optimization: Distributed CSP/Optimization
• Data Mining & Knowledge Management: Representing, Reasoning, and Using Provenance, Trust

Application Category

• Security and Privacy: Data transparency and provenance
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Data lakehouses (LHs) are at the core of current cloud analytics stacks by providing elastic, relational compute on data in cloud data lakes across vendors. For relational semantics, they rely on open table formats (OTFs). Unfortunately, they have many missing features inherent to their metadata designs, like no support for multi-table transactions and recovery in case of an abort in concurrent, multi-query workloads. This, in turn, can lead to non-repeatable reads, stale data, and high costs in productive cloud systems. In this work, we introduce LakeVilla, a complementary solution that introduces recovery, multi-query/table transactions, and transaction isolation to state-of-the-art OTFs like Apache Iceberg and Delta Lake tables. We investigate its transactional guarantees and show it has minimal impact on performance (2% YCSB writes, 2.5% TPC-DS reads) and provides concurrency control for multiple readers and writers for arbitrary long transactions in OTFs in a non-invasive way.

Problem

• Enables multi-table transactions in lakehouses
• Provides recovery for aborted concurrent workloads
• Ensures transaction isolation without performance penalty

Innovation

• Introduces recovery for multi-table transactions
• Provides transaction isolation in open table formats
• Minimal performance impact on read and write operations