Coded Computing for Dynamic System via a Cartesian Product

2026-09-30

AI Summary

This study addresses task execution bottlenecks in dynamic MapReduce systems caused by node departures or cluster joinings, proposing a general coded distributed computing scheme. The core methodology introduces a Cartesian-product-based coupling mechanism of cluster-level placement delivery arrays (PDAs) alongside delivery-aware reallocation rules. By leveraging the cached data of surviving nodes and the storage of newly joined clusters, this approach enables non-Reduce-capable nodes to function as coded transmitters, thereby accommodating dynamic topology changes. Furthermore, the exact communication load under arbitrary disconnection scenarios is theoretically derived. Performance analysis demonstrates that the proposed scheme approaches optimality in the absence of disconnections, while achieving approximation ratios of 2 and 4 under specific multicast classes and global optimization, respectively, when disconnections occur.

Abstract

This paper studies coded distributed computing (CDC) in a dynamic system in which workers may depart and new clusters may join. The caches of the surviving workers and their existing Reduce assignments stay untouched, while the storage brought by arriving clusters is put to use. In contrast to elastic computing, which targets linear functions, and to dynamic coded caching, which requires placement across all users, the model considered here accommodates general MapReduce tasks under a placement that is partly fixed and partly new. The proposed scheme builds cluster-wise placement delivery arrays (PDAs) and couples them through a Cartesian product. A delivery-aware rule reassigns the abandoned Reduce functions, and a generalized communication PDA allows even workers that hold no Reduce function to serve as coded transmitters. For arbitrary feasible disconnections, the exact load is derived. A file-wise converse for instantly decodable XOR multicasts shows that, in the absence of disconnections, the scheme lies within a factor of two of the best scheme in this multicast class when new clusters arrive, and within a factor of four otherwise, even when the benchmark optimizes over all feasible uncoded placements.

Problem

• Coded Distributed Computing
• Dynamic System
• MapReduce
• Placement Delivery Array

Innovation

• Coded Distributed Computing
• Placement Delivery Array
• Cartesian Product
• MapReduce
• Dynamic Systems