Unifying In-Memory Data Analytics through Sparse Compilation

2026-09-24

AI Summary

This study addresses the challenge of efficiently executing heterogeneous data analytics workloads on multicore hardware by proposing Reffine, a novel execution engine. Reffine introduces a unified abstraction grounded in relational algebra and sparse iteration theory. Through an innovative intermediate representation, it enables operator fusion and automatic parallelization, subsequently compiling these into hardware-efficient sparse code to support workload-agnostic end-to-end optimization. Experimental evaluations demonstrate that Reffine achieves speedups of 24.9× and 3.2× over DuckDB and Umbra, respectively, on the TPC-H benchmark. Furthermore, in streaming and graph analytics scenarios, it outperforms Polars and NetworkX by 18.3× and 47.9×, respectively, highlighting its effectiveness across diverse analytical domains.

Abstract

As modern data analytics workloads become increasingly heterogeneous and hardware-intensive, achieving efficient multi-core performance across diverse applications remains an open challenge. We present Reffine, a compiler-based in-memory analytics engine that delivers high performance across a broad range of data analytics workloads. Reffine introduces a novel intermediate representation (IR), grounded in relational algebra and sparse iteration theory, that provides a unified abstraction for data and computation. This representation enables workload-agnostic, end-to-end optimizations such as operator fusion and automatic parallelization across diverse analytics applications. We further develop a sparse compiler backend that translates Reffine IR into hardware-efficient imperative code, achieving high multi-core performance without domain-specific implementations. On the TPC-H benchmark, Reffine outperforms the in-memory analytical database DuckDB by up to $24.9\times$ and the state-of-the-art compilation-based database Umbra by up to $3.2\times$. Reffine also achieves average speedups of $18.3\times$ and $47.9\times$ over Polars and NetworkX on streaming and graph analytics workloads, respectively. Source code: https://github.com/ampersand-projects/reffine

Problem

• data analytics
• multi-core performance
• heterogeneous workloads
• in-memory analytics

Innovation

• Sparse Compilation
• Intermediate Representation
• In-Memory Analytics
• Operator Fusion
• Automatic Parallelization