MDTP -- An Adaptive Multi-Source Data Transfer Protocol

2025-05-14

AI Summary

To address efficiency bottlenecks—namely server load imbalance and response asynchrony—in multi-source parallel scientific data transfer under rapidly growing data volumes, this paper proposes an adaptive chunking-based dynamic scheduling mechanism. We formulate file requests as an enhanced bin-packing problem and jointly optimize chunk granularity, source selection, and scheduling order to align per-round completion times across servers and achieve load balancing. The method integrates performance-aware dynamic chunk-size allocation, multi-source parallel downloading, and real-time bandwidth adaptation. Experiments demonstrate a 10–22% throughput improvement over Aria2, significantly outperforming static chunking and BitTorrent. Moreover, the approach maintains high throughput stability under high network latency or degraded primary-node bandwidth, thereby enhancing robustness and resource utilization in heterogeneous network environments.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Search and Optimization: Distributed Search
• Planning, Routing, and Scheduling: Scheduling under Uncertainty

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems

Abstract

Scientific data volume is growing in size, and as a direct result, the need for faster transfers is also increasing. The scientific community has sought to leverage parallel transfer methods using multi-threaded and multi-source download models to reduce download times. In multi-source transfers, a client downloads data from multiple replicated servers in parallel. Tools such as Aria2 and BitTorrent support such multi-source transfers and have shown improved transfer times. In this work, we introduce Multi-Source Data Transfer Protocol, MDTP, which further improves multi-source transfer performance. MDTP logically divides a file request into smaller chunk requests and distributes the chunk requests across multiple servers. Chunk sizes are adapted based on each server's performance but selected in a way that ensures each round of requests completes around the same time. We formulate this chunk-size allocation problem as a variant of the bin-packing problem, where adaptive chunking efficiently fills the available capacity"bins"corresponding to each server. Our evaluation shows that MDTP reduces transfer times by 10-22% compared to Aria2, the fastest alternative. Comparisons with other protocols, such as static chunking and BitTorrent, demonstrate even greater improvements. Additionally, we show that MDTP distributes load proportionally across all available replicas, not just the fastest ones, which improves throughput. Finally, we show MDTP maintains high throughput even when latency increases or bandwidth to the fastest server decreases.

Problem

• Improving multi-source data transfer performance
• Adaptive chunk allocation for balanced server load
• Reducing transfer times compared to existing protocols

Innovation

• Adaptive multi-source data transfer protocol
• Dynamic chunk-size allocation for servers
• Improved throughput with proportional load distribution