file format design

Existing learned indexes struggle to simultaneously achieve high concurrency, durability, and low intrusiveness under write-intensive workloads. This work proposes a hierarchical learned indexing architecture that leverages the separation between Memtables and SST files in RocksDB to enable targeted optimizations at both memory and disk layers. By reusing structural knowledge across Memtables, the approach mitigates the overhead of frequent index reconstruction, while a block-aware, read-only learned index ensures that lookups complete within a single I/O in the worst case—without requiring modifications to the storage layer or read path. Experimental results demonstrate that, across diverse large-scale workloads, the proposed method improves write throughput by up to 1.5× and read throughput by up to 2.1× compared to state-of-the-art systems.

Existing LSM-tree and B-tree–based KV stores—as well as systems like FASTER—face bottlenecks in index memory overhead, log-compaction efficiency, and hot working-set management for large-scale KV services characterized by memory constraints, highly skewed access patterns, and datasets far exceeding main memory capacity. Method: This paper proposes a hierarchical record-oriented KV storage architecture featuring (i) a novel multi-threaded lock-free log compaction mechanism, (ii) a lock-free read-modify-write (RMW) algorithm, (iii) a two-level hash index coupled with a read cache, and (iv) tight integration of LSM-style cold-hot data separation with a CPU-optimized lock-free concurrency engine, specifically tailored to NVMe SSD characteristics. Results: Under typical skewed workloads, the system achieves 2.1×–11.75× higher average throughput than RocksDB, significantly reduces write amplification, and delivers low latency with high robustness.

High-dimensional approximate nearest neighbor (ANN) indexes suffer from redundancy and high search overhead when serialized into generic file formats. Method: We propose eCP-FS—the first disk-based ANN indexing system that directly models the index as a human-readable, program-parsable filesystem structure, replacing conventional serialization with filesystem abstractions. Built upon the eCP indexing algorithm and standard filesystem libraries, eCP-FS enables cross-language interoperability and transparent access. Contribution/Results: Experiments show eCP-FS achieves minimal memory footprint under memory-constrained settings, with pronounced advantages in multi-index coexistence scenarios; it remains competitive even with ample memory. This work validates the feasibility and practicality of the “file-as-index” paradigm, substantially reducing debugging and maintenance complexity for ANN indexes.

This work addresses the inefficiency of conventional storage formats in supporting high-throughput, randomly shuffled reads and compact distribution of complete molecular records for atomistic machine learning. To overcome these limitations, the authors propose Atompack, a novel append-optimized storage and distribution format that treats entire molecules as atomic units. Atompack employs an immutable index and memory-mapped I/O, eliminating field-wise chunking and runtime record reconstruction. Experimental results on a representative task involving 64-atom systems demonstrate that Atompack achieves a 96× speedup in shuffled read performance and a 79% reduction in data volume compared to ASE LMDB, substantially improving both training data construction efficiency and access performance.

Compaction in LSM-trees is essential but incurs substantial I/O overhead and significant write/read amplification; moreover, conventional approaches decouple data transformations—such as column-family merging, format conversion, and index construction—from compaction, leading to redundant I/O and increased latency. This paper proposes Transformation-Embedded LSM-tree (TE-LSM), the first design to deeply integrate general-purpose user-defined transformations into the compaction pipeline, enabling semantically consistent data reorganization and transformation concurrently during I/O-intensive compaction. TE-LSM introduces a compaction-aware cost model, cross-column-family merging, column-group splitting, and incremental indexing. Implemented transparently atop RocksDB, it achieves efficient, end-to-end integration. Experiments show only a 20% write-throughput reduction—significantly better than external transformation schemes (35–60% degradation)—and up to 425% read-latency reduction, markedly improving end-to-end efficiency and query readiness.

This work addresses the high analytical latency and severe resource contention in HTAP systems caused by traditional ETL pipelines, which entail frequent data movement. To overcome these limitations, the authors propose offloading data transformation logic to an intelligent storage layer that leverages near-data or in-storage computing capabilities to perform format conversion and preprocessing directly at the storage tier. This approach eliminates the overhead of data migration and significantly reduces interference with foreground transactional workloads, thereby enhancing both the performance and real-time responsiveness of analytical queries. Experimental results demonstrate that the proposed architecture achieves a highly reusable, low-latency data processing paradigm under mixed workloads across multiple execution engines, offering an efficient and scalable storage-compute co-design for HTAP systems.

In SSD-backed approximate nearest neighbor (ANN) search, I/O overhead accounts for 70%–90% of total query cost, severely limiting performance. This work systematically investigates the joint optimization space across memory layout, disk layout, and search algorithms for graph-based disk ANN systems. We introduce a page-level complexity model to quantify how page locality and path length impact I/O, revealing cross-dimensional synergies. Building on these insights, we design OctopusANN, a high-performance system that integrates memory-resident navigation, dynamic beam width, page reordering, and intra-page search strategies, all deeply tailored to graph index structures and SSD characteristics. Experimental results show that at Recall@10 = 90%, OctopusANN achieves 4.1%–37.9% higher throughput than Starling and 87.5%–149.5% higher throughput than DiskANN, significantly reducing I/O overhead.

To address the high energy consumption and latency of in-memory database search under high raw bit error rates (BER), this paper proposes an in-memory computing SQL processing architecture leveraging hyperdimensional computing (HDC) and ferroelectric NAND (FeNAND). We innovatively design an HDC encoding scheme tailored for relational tables, mapping SQL predicate filtering and aggregation operations to high-dimensional vector similarity computations. By exploiting FeNAND’s multi-level cell (MLC) capability, our architecture enables bit-level parallel in-memory computation without explicit error correction, tolerating up to 10% storage errors robustly. Evaluated on TPC-DS fact tables, our approach achieves up to 80.6× lower latency and 12,636× lower energy consumption compared to state-of-the-art CPU- and GPU-based database engines. This work presents the first demonstration of robust and efficient in-situ database query execution on highly noisy, non-volatile memory.

This work addresses the I/O bottleneck in disk-resident graph-based approximate nearest neighbor (ANN) search under memory-constrained settings, where existing approaches lack a systematic understanding of storage, layout, and execution strategies. The authors propose a unified taxonomy that decomposes such systems into five core components: storage strategy, disk layout, cache management, query execution, and update mechanism. Through fine-grained evaluation and end-to-end experiments, they uncover several non-intuitive insights: vector dimensionality significantly impacts component effectiveness; current disk layouts exhibit I/O utilization consistently below 15%; smaller page sizes outperform larger ones under optimized layouts; and update strategies must be tailored to specific workloads. These findings provide empirical foundations and practical design guidelines for building efficient disk-based ANN systems.

This work addresses the high latency introduced by traditional decompression in scientific data analysis, which undermines the storage and transmission benefits of compression. To overcome this limitation, the authors propose a multi-stage, error-bounded decompression and homomorphic analysis framework. By abstracting a generic compression pipeline, the framework enables hierarchical partial decompression and introduces homomorphic operation algorithms tailored to three representative scientific analysis tasks, allowing computations to be performed directly on intermediate compressed representations without full decompression. Implemented atop four mainstream compressors and evaluated across five real-world datasets, the approach consistently reduces data access latency and significantly improves analytical efficiency across diverse workloads.