Modular Architecture for High-Performance and Low Overhead Data Transfers

2025-08-07

AI Summary

To address low efficiency and poor stability in large-scale data transfers within geo-distributed environments, this paper proposes AutoMDT, a modular data transfer architecture. AutoMDT decouples I/O and network tasks and introduces, for the first time, a Proximal Policy Optimization (PPO)-based deep reinforcement learning agent to jointly optimize concurrency levels across read, network, and write stages. A lightweight network-system co-simulator enables offline training, ensuring rapid convergence and dynamic adaptability to heterogeneous and time-varying network conditions. Evaluated on a production-grade platform, AutoMDT reduces average transfer completion time by 68% compared to state-of-the-art baselines, accelerates convergence by up to 8×, and significantly improves resource utilization and transfer robustness under network fluctuations and workload variability.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Transfer, Domain Adaptation, Multi-Task Learning

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Economics, Online Markets and Human Computation: Incentives in network design for Web infrastructures and ecosystems
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

High-performance applications necessitate rapid and dependable transfer of massive datasets across geographically dispersed locations. Traditional file transfer tools often suffer from resource underutilization and instability because of fixed configurations or monolithic optimization methods. We propose AutoMDT, a novel modular data transfer architecture that employs a deep reinforcement learning based agent to simultaneously optimize concurrency levels for read, network, and write operations. Our solution incorporates a lightweight network-system simulator, enabling offline training of a Proximal Policy Optimization (PPO) agent in approximately 45 minutes on average, thereby overcoming the impracticality of lengthy online training in production networks. AutoMDT's modular design decouples I/O and network tasks, allowing the agent to capture complex buffer dynamics precisely and to adapt quickly to changing system and network conditions. Evaluations on production-grade testbeds show that AutoMDT achieves up to 8x faster convergence and a 68% reduction in transfer completion times compared with state-of-the-art solutions.

Problem

• Optimizing high-performance data transfers for large datasets
• Overcoming resource underutilization in traditional transfer tools
• Adapting dynamically to changing system and network conditions

Innovation

• Modular architecture decouples I/O and network tasks
• Deep reinforcement learning optimizes transfer operations
• Lightweight simulator enables fast offline PPO training