columnar storage design

Log-structured table formats (e.g., Delta Lake, Iceberg, Hudi) in data lakes suffer from excessive small files due to append-only writes and metadata-heavy operations, degrading query performance, increasing storage costs, and limiting system scalability. Existing compaction mechanisms lack flexibility, pursue narrow objectives, and fail to balance benefits against operational overhead. To address this, we propose Scalable Adaptive Compaction (SAC), an extensible, workload-aware, metadata-driven framework featuring dynamic threshold tuning, lightweight online evaluation, and a modular rule engine. SAC is production-deployed via the OpenHouse control plane. Evaluated on LinkedIn’s production workloads and synthetic benchmarks, SAC reduces file counts by up to 92%, improves typical query latency by 3.8×, and maintains bounded runtime overhead.

To address the poor update performance of LSM-tree-based columnar storage under mixed workloads, this paper proposes a novel hybrid row-column storage engine: it maintains an in-memory incremental row store for efficient real-time updates, and upon compaction, applies fine-grained row-to-column conversion and asynchronous compression to jointly optimize update throughput and query efficiency. Key contributions include: (1) the first integrated architecture synergizing incremental row storage with columnar storage; (2) a fine-grained conversion mechanism coupled with adaptive columnar encoding; and (3) a cost-aware background resource scheduling strategy. Experimental evaluation demonstrates that, under mixed workloads, the engine achieves significantly higher update throughput than state-of-the-art columnar systems (e.g., DuckDB), while sustaining high query performance—effectively overcoming the long-standing real-time update bottleneck in columnar storage.

Existing Computational Object Storage (COS) systems face three key bottlenecks when performing large-scale scientific tabular data SQL analytics in HPC environments: rigid output formats, limited operator pushdown capability, and inadequate adaptation to deep storage hierarchies. To address these, we propose COS-SQL—a near-data SQL analytics framework tailored for HPC. Our approach features: (1) flexible output format support—including Arrow columnar layout; (2) full-stage pushdown of complex operators and array expressions; and (3) dynamic execution path selection based on hierarchical storage structure. COS-SQL adopts object-level storage organization and tightly integrates with Apache Spark. Evaluated on real-world HPC workloads, it achieves up to 32.7% end-to-end performance improvement over state-of-the-art COS systems, significantly enhancing both analytical flexibility and execution efficiency.

To address the high communication overhead of multilingual persistent operations and the suboptimal cross-model processing efficiency caused by monolithic storage engines in existing multimodel databases, this paper proposes an integrated multimodel storage engine architecture. The architecture unifies heterogeneous storage engines, each specialized and optimized for a distinct data model; introduces a multi-stage hash join algorithm to enable efficient cross-model joins; and implements unified query plan compilation and coordinated execution across models. Experimental evaluation demonstrates that the system achieves up to 188× speedup over the best-performing baseline on representative multimodel analytical workloads, while significantly improving both performance and scalability under complex, mixed-model query loads.

This work investigates how modern database systems can achieve low-overhead, high-throughput I/O via the Linux io_uring interface. Focusing on two representative workloads—storage-intensive buffer management and network-intensive analytical processing—we propose and empirically validate systematic design principles for integrating io_uring into databases. Our methodology centers on three key mechanisms: persistent buffer registration, passthrough I/O, and a unified I/O–network programming model, rigorously analyzing their end-to-end performance impact. Innovatively, we co-design asynchronous batched I/O, zero-copy passthrough, and architecture-level coordination to establish a portable I/O optimization framework. We implement this approach in PostgreSQL and demonstrate a 14% throughput improvement under realistic workloads. To our knowledge, this is the first systematic study to empirically confirm that fine-grained, kernel-level I/O primitive optimizations yield substantial, measurable gains at the full-system level.

This study addresses the challenge of selecting an optimal Data Lakehouse architecture based on data type and scale by presenting the first systematic evaluation of Apache Hudi, Apache Iceberg, and Delta Lake in terms of data ingestion efficiency and storage overhead for structured and semi-structured workloads. Conducted on the Apache Spark platform, the empirical comparison employs a four-stage ETL pipeline to assess the three frameworks under realistic conditions. Experimental results demonstrate that Delta Lake achieves the fastest data loading performance, while Iceberg excels in storage compression ratio and system stability. In contrast, Hudi exhibits comparatively lower efficiency in both batch ingestion and storage utilization. These findings provide critical empirical evidence and practical guidance for informed architectural decisions in Lakehouse deployments.

This study addresses the lack of systematic methodologies for selecting data architectures in modern organizations grappling with vast, heterogeneous data environments. To this end, it proposes the DATER conceptual framework, which establishes a unified taxonomy of technical requirements and systematically examines the historical evolution, core characteristics, and applicability boundaries of six prominent data architectures: data warehouses, data lakes, lakehouses, data fabrics, and data meshes. Through conceptual modeling and multidimensional comparative analysis, the framework clarifies overlaps and distinctions among these architectures, articulating their respective strengths and limitations. By offering a structured evaluation tool, DATER significantly enhances the strategic alignment and contextual appropriateness of data architecture design for both researchers and practitioners.

This study addresses the performance degradation in lakehouse tables caused by small-file accumulation, a problem exacerbated by the absence of principled merge-triggering mechanisms. The authors develop an open simulation framework that generates diverse table layouts based on Apache Iceberg and extracts 17 metadata features to predict post-merge file reduction ratios with high accuracy using XGBoost (R²=0.998). Their work is the first to systematically uncover strong correlations between metadata characteristics and merge efficacy. They further propose a model-free merging strategy relying solely on a single partition-level threshold, which demonstrates remarkable generalization across heterogeneous workloads (R²=0.976). Experimental results show that this approach substantially improves metadata-intensive query performance while also revealing its subtle impact on the parallelism of full-table scan compaction.

本文针对基因组和宏基因组分析中数据移动和准备的瓶颈问题，提出了一种以存储为中心的设计方案，通过在存储系统内部进行数据分析并支持高度压缩的数据存储，从而提高性能、能效及成本效益。

This study addresses the challenge organizations face in selecting among the three dominant data warehouse modeling approaches—Inmon, Kimball, and Data Vault—by developing a multidimensional evaluation framework. The framework systematically compares these methodologies across key dimensions including architectural philosophy, modeling techniques, scalability, agility, query performance, and auditability. Integrating contextual factors such as organizational size, regulatory compliance requirements, and analytical maturity, the work proposes a situational selection guideline that aligns modeling choices with strategic business objectives rather than purely technical merits. By demonstrating that no single approach is universally optimal and articulating a practical, context-sensitive decision-making standard, this research fills a critical gap in practitioner-oriented guidance for data warehouse design.