durable storage design

Modern data storage systems suffer from latent cross-layer faults due to tight hardware–software coupling across multiple abstraction layers, often leading to silent data corruption or unrecoverable data loss. To address this, we propose the first cross-layer fault-tolerance analysis framework targeting heterogeneous storage stacks—including SSDs, persistent memory, local file systems, and distributed storage. Our approach combines architectural modeling of the full stack, systematic injection of representative defects, and precise tracking of fault propagation across hardware–firmware–software boundaries to expose error propagation paths and consistency violation mechanisms. Through empirical evaluation across widely deployed systems, we identify critical vulnerabilities impacting data integrity and quantify coverage gaps in existing fault-tolerance techniques. The framework provides a scalable, principled methodology for analyzing cross-layer resilience and establishes concrete, actionable directions for designing next-generation highly reliable storage systems.

This paper addresses the underappreciated role of persistence in large-memory systems, asserting that “persistence must be treated as a first principle of system design.” It systematically analyzes structural challenges arising from vertical (capacity/latency) and horizontal (network flattening) scaling of the memory hierarchy. The authors propose a full-stack co-designed persistence paradigm: (1) elevating persistence to the foundational axiom of system architecture, and (2) introducing a dual-mechanism persistence model—predictable speculative persistence for performance and strongly consistent deterministic persistence for correctness. By enabling mobile persistence and jointly optimizing memory, interconnect, and storage layers, the approach achieves high throughput, low latency, and cost efficiency simultaneously. Extensive evaluations across diverse workloads demonstrate significant improvements in both system performance and reliability.

Fundamental trade-offs between availability and consistency in distributed storage are often oversimplified in existing work—particularly through extreme models like CAP—lacking precise characterization of how object semantics interact with diverse consistency models. Method: We propose a unified semantic framework that integrates operational semantics with multi-level consistency constraints—including causal consistency, sequential consistency, snapshot isolation, and SQL transactional semantics—covering key data objects such as key-value stores, counters, sets, CRDTs, and transactional databases. Contribution/Results: We introduce the Arbitration-Free Consistency (AFC) theorem, establishing that an object admits highly available implementations if and only if its visibility or read dependencies do not require total-order arbitration. This theorem unifies and generalizes classical results including CAP, providing the first formal, object-centric criterion for designing highly available distributed systems.

This work addresses the joint optimization of media selection, capacity allocation, and data placement (replication vs. tiering) for key-value caching across heterogeneous NVM/DRAM/disk storage under memory budget constraints. We introduce the first systematic modeling framework for multi-level non-volatile cache configurations, analytically characterize the operational regimes where replication or tiering dominates, and propose an adaptive configuration policy grounded in device failure rates and data update frequencies. Our methodology integrates cache access behavior modeling, hierarchical configuration optimization, and empirical validation using memcached benchmarks. Results demonstrate that tiering substantially outperforms replication under low device failure rates and high update workloads. Key contributions include: (1) a deployable, low-overhead configuration algorithm; (2) quantitative design guidelines for heterogeneous cache deployment; and (3) theoretical foundations for the reliability–performance trade-off in tiered caching systems.

In digital forensics, the atomicity and integrity of storage snapshots lack rigorous definitions that jointly guarantee both instantaneousness and causal ordering—undermining evidentiary admissibility in legal proceedings. To address this, we propose a novel atomicity definition grounded in causal consistency, overcoming the limitation of conventional time-based atomicity models. We further rectify conceptual flaws in existing integrity definitions and introduce a revised, theoretically sound yet engineering-practical integrity criterion—explicitly supporting copy-on-write (CoW) implementations. Our approach integrates causal modeling, formal snapshot semantics, CoW mechanism analysis, and formalization of forensic quality criteria, yielding a verifiable snapshot semantic framework. This work establishes the first theoretical foundation for forensic tool design that unifies causal ordering with instantaneous state capture, thereby significantly enhancing the forensic validity and judicial admissibility of live data acquisition.

This work demonstrates that the atomicity assumptions underpinning traditional Unix tools do not hold under crash consistency, leading to pervasive data corruption and system failures. It introduces the Forward-in-Time Ordering (FITO) analysis framework, which formally proves—for the first time—that no layer in the storage stack, from CPU to persistent media, achieves true atomic persistence. Apparent atomicity arises instead from implicit, cross-layer timing assumptions that leak across abstraction boundaries. System-call-level persistence primitives fail to define a well-defined commit boundary under crashes, resulting in a recursive chain of non-atomic dependencies—a structural flaw inherent to the entire stack. Through formal methods and cross-layer empirical evaluation spanning ext4, NVMe, Linux reboot semantics, and x86-64 architecture, this study identifies FITO assumption violations as the root cause of large-scale cloud outages, database corruptions, wasted AI training cycles, and financial losses, all amplified by retry-induced error propagation.

Existing non-volatile memory (NVM)-based key-value stores commonly lack support for modern database interfaces such as snapshots, consistent iterators, and atomic batch operations. To address this limitation, this work proposes and implements FlintKV, a skip-list-based NVM-optimized storage engine that natively supports a full production-grade transactional API while guaranteeing persistent linearizability. FlintKV achieves this through a novel concurrency protocol that integrates multi-version concurrency control with flat combining, leveraging byte-addressable NVM, custom persistence mechanisms, and highly efficient data structures. Experimental results demonstrate that FlintKV significantly enhances performance, achieving up to 75% higher end-to-end throughput compared to state-of-the-art alternatives. The system can be deployed standalone or integrated as a functional enhancement module into existing NVM-based storage systems.

Persistent memory programs often sacrifice performance due to excessive use of flush and fence instructions, making it challenging to balance crash consistency with hardware efficiency. This work proposes a black-box binary rewriting technique that requires neither source code nor manual intervention. By leveraging semantic analysis and performance modeling, the method automatically identifies and optimizes redundant persistence instruction sequences while strictly preserving crash consistency semantics. Experimental evaluation on multiple real-world persistent memory applications demonstrates performance improvements of up to 15%, marking the first fully automated and efficient optimization of synchronization instructions for persistent memory programs.

Shared state profoundly influences the performance and fault tolerance of stream processing, service-oriented, and continual learning systems, yet existing approaches often treat access control, hardware-aware execution, memory management, and long-term evolution in isolation. This work reframes state management as a runtime control problem and introduces a contract-driven blueprint centered on state objects, control planes, coupling paths, evaluation boundaries, and pending contracts. Building upon this foundation, we develop a unified analytical framework encompassing state-access scheduling, state-aware execution, and state evolution reuse. Through systematic scheduling, runtime control, and cross-layer coupling analysis, our approach identifies critical anti-patterns and advances a perturbation-aware evaluation paradigm, thereby establishing both theoretical foundations and practical design guidelines for state control in distributed systems.