design distributed storage

To address inefficient data migration and inaccurate performance prediction in heterogeneous storage systems (NVMe cache + HDD backend), this paper designs and implements a distributed two-tier storage system. We propose an online reinforcement learning–based dynamic data tiering scheduling algorithm and develop an end-to-end performance model integrating queuing network theory with fine-grained device behavior modeling. Our key contribution is the first scalable, fine-grained device behavior modeling method tailored for heterogeneous storage—enabling adaptive tiering management and precise performance prediction under high-concurrency I/O workloads in multi-core clusters. Experimental evaluation on multi-node clusters demonstrates an average model prediction error of less than 8%, a 27% improvement in I/O throughput, and a 34% reduction in average access latency. The framework provides a reusable modeling and optimization foundation for two-tier storage systems.

This study addresses the practical disparities and co-evolution between high-performance computing (HPC) and edge computing architectures within the cloud continuum. It presents the first large-scale empirical analysis based on 396 real-world, production-grade AWS architectures. Methodologically, we propose a multidimensional, data-driven framework encompassing service topology identification, storage type classification, architectural complexity quantification, and ML service integration statistics. Results reveal systematic differences—and complementary patterns—between HPC and edge architectures across four dimensions: core service composition (e.g., EC2 versus Greengrass/Lambda), storage design paradigms (parallel file systems versus distributed lightweight caches), complexity distributions, and ML embedding strategies. This work delivers the first industry-scale architectural benchmark for the cloud continuum, providing empirically grounded insights and methodological foundations for cross-domain architecture design, resource optimization, and cloud-native convergence of HPC and edge computing.

Distributed file systems (DFS) exhibit divergent fault tolerance and horizontal scalability characteristics, yet systematic, cross-architecture evaluation under realistic hybrid workloads remains scarce. Method: We conduct a comprehensive empirical study of Google File System (GFS), Hadoop Distributed File System (HDFS), and MinIO—representing legacy, big-data, and cloud-native paradigms—using unified benchmarks on physical clusters. We apply stress testing, controlled fault injection, and protocol-level log analysis to quantify throughput, recovery latency, and consistency guarantees across data redundancy, node failure recovery, and high-concurrency client access. Contribution/Results: This is the first study to comparatively evaluate these three architecturally distinct DFS under concurrent cloud-native and big-data workloads. We identify MinIO’s low-latency advantage for small files, HDFS’s stability limits in batch processing, and GFS’s enduring influence on lightweight DFS design. Based on these findings, we propose a scenario-driven selection framework—distinguishing high-availability storage from analytical workloads—to guide industrial storage system deployment with empirical evidence.

With AI and high-resolution simulations increasingly driving HPC workloads, parallel I/O performance bottlenecks have grown more complex, while existing optimization tools remain fragmented and difficult to select. Method: We systematically review 131 publications and—employing bibliometric analysis, systematic literature review, and taxonomy modeling—construct the first comprehensive, end-to-end parallel I/O classification framework (a “360° taxonomy”) covering characterization, analysis, and optimization. Our approach integrates cross-platform profiling and tracing tools—including Darshan, Vampir, and Lustre trace—into a unified analytical pipeline. Contribution: We propose the first holistic, cross-layer I/O optimization framework spanning applications, runtime systems, file systems, and hardware; release a structured knowledge graph and open-source classification toolkit; and significantly reduce decision-making overhead in selecting optimization strategies. This work delivers a reusable, scalable methodology for enhancing parallel I/O performance in production HPC environments.

This work addresses the joint optimization of media selection, capacity allocation, and data placement (replication vs. tiering) for key-value caching across heterogeneous NVM/DRAM/disk storage under memory budget constraints. We introduce the first systematic modeling framework for multi-level non-volatile cache configurations, analytically characterize the operational regimes where replication or tiering dominates, and propose an adaptive configuration policy grounded in device failure rates and data update frequencies. Our methodology integrates cache access behavior modeling, hierarchical configuration optimization, and empirical validation using memcached benchmarks. Results demonstrate that tiering substantially outperforms replication under low device failure rates and high update workloads. Key contributions include: (1) a deployable, low-overhead configuration algorithm; (2) quantitative design guidelines for heterogeneous cache deployment; and (3) theoretical foundations for the reliability–performance trade-off in tiered caching systems.

Scheduling data-intensive workloads in large-scale distributed systems faces challenges including complexity, heterogeneous parallelism, data locality constraints, and multi-dimensional QoS optimization (e.g., timeliness, fault tolerance, energy efficiency). Method: This paper proposes a novel workload classification scheme grounded in data characteristics and service requirements; systematically surveys and structures mainstream scheduling strategies, exposing critical limitations in dynamic adaptability, fine-grained fault tolerance, and energy–QoS co-optimization; and introduces a unified scheduling framework integrating data-locality awareness, elastic parallel scheduling, QoS-tiered guarantees, and energy-aware resource allocation. Contribution/Results: The study establishes a scalable classification paradigm, delivers a clear technology evolution roadmap, and identifies a prioritized list of open research challenges—thereby advancing foundational understanding and guiding future design of intelligent, holistic schedulers for modern distributed data systems.

Traditional distributed file systems struggle to handle fine-grained, dynamically varying workloads due to coarse-grained resource reservation and scaling mechanisms, often resulting in resource underutilization and increased response latency. This work proposes the first fully serverless distributed file system, in which both data and metadata operations are delegated to short-lived, multithreaded functions. A policy-driven coordinator dynamically maps files to function instances, enabling fine-grained elasticity and cost efficiency. The design substantially mitigates cold-start overhead—reducing it by up to 580× compared to baseline approaches—and achieves comparable or superior performance while lowering costs by 63% relative to λFS, 68% relative to Amazon EFS, and 63% relative to Ceph.

To address high latency and cost in geo-distributed cloud environments caused by graph data’s topological dependencies and localized access patterns, this paper proposes a hierarchical graph storage framework that jointly optimizes replica placement and request routing. Our approach introduces: (1) a latency-aware hierarchical graph structure to reduce decision complexity and mitigate network heterogeneity; (2) an overlap-aware replica placement strategy to improve coverage efficiency for critical subgraphs; (3) a directed hot-diffusion model for dynamic data allocation; and (4) a layer-wise expansion routing algorithm tailored to graph-pattern access characteristics. Experimental results demonstrate that the framework achieves 1.34×–3.67× speedup in online graph query response time and 1.28×–3.56× acceleration in offline graph analytics performance, significantly outperforming state-of-the-art baselines.

Modern storage hierarchies face a fundamental trade-off between load balancing and space efficiency. To address this, we propose Mirror-Optimized Storage Tiering (MOST), a co-design strategy integrating mirroring with tiered storage. MOST implements dynamic hot-data identification and cross-tier mirroring via Cerberus—a user-space storage management layer built atop CacheLib—thereby eliminating the high-overhead data migrations inherent in conventional tiering. Its core innovation lies in employing lightweight mirroring to enhance I/O parallelism and bandwidth utilization while preserving the space efficiency of tiered storage. Experimental evaluation across diverse I/O-intensive and dynamic workloads demonstrates that Cerberus achieves an average 32% throughput improvement over state-of-the-art approaches; gains are especially pronounced in NVMe+SSD hybrid tiers.

This study addresses the lack of systematic optimization in cloud data pipelines with respect to cost, execution time, and resource utilization, particularly in multi-tenant and industrial settings where research remains limited. Through a comprehensive systematic literature review, the work establishes a unified classification framework for optimization objectives that encompasses both single- and multi-cloud environments as well as batch and stream processing paradigms. The analysis synthesizes existing approaches and identifies critical research gaps, including insufficient support for multi-tenancy, inadequate multi-cloud coordination, and a scarcity of real-world deployment validation. By clarifying the core objectives and technical pathways for optimizing cloud data pipelines, this paper provides a theoretical foundation and clear direction for future research in this domain.