Optimizing High-Throughput Distributed Data Pipelines for Reproducible Deep Learning at Scale

2026-04-23

AI Summary

This study addresses the low GPU utilization and irreproducibility commonly encountered in large-scale distributed GPU training due to data loading bottlenecks. Through a systematic analysis of I/O and CPU constraints, the authors propose an optimized architecture that integrates push-based worker transformation with local disk caching. They design a Fanout-Cache mechanism and a dedicated polling queue to eliminate race conditions arising from multi-process shared queues, and incorporate deterministic random number generation to ensure strict reproducibility in high-throughput data pipelines—a first in this context. Leveraging optimizations based on Petastorm, Parquet, and PyArrow-to-NumPy conversions, the approach reduces end-to-end training time from 22 hours to 3 hours (a 6× speedup), increases GPU utilization to over 60%, and substantially decreases inter-run variance.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems

Application Category

• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• Security and Privacy: Data transparency and provenance
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

Training massive-scale deep learning models on datasets spanning tens of terabytes presents critical challenges in hardware utilization and training reproducibility. In this paper, we identify and resolve profound data-loading bottlenecks within distributed GPU training pipelines using the Petastorm data loader and Apache Parquet datasets. Through systematic profiling, we demonstrate that network I/O and CPU-bound data transformations (e.g., PyArrow to NumPy) constrain GPU utilization to as low as 10-15%. To address this, we propose an optimized architecture that features push-down worker-level transformations coupled with local-disk caching via Fanout-Cache, minimizing redundant I/O and CPU overhead across training epochs. Furthermore, we eliminate race conditions in multi-worker shared queues by implementing dedicated round-robin ventilator and result queues, alongside modernized RNG handling, achieving strict deterministic data loading. Our optimizations yield a 6x speedup, reducing end-to-end training time from 22 hours to 3 hours, increasing GPU utilization to over 60%, and drastically reducing run-to-run variance, enabling robust, high-throughput, and reproducible large-scale model training.

Problem

• data-loading bottleneck
• training reproducibility
• GPU utilization
• distributed training
• large-scale deep learning

Innovation

• data pipeline optimization
• deterministic data loading
• distributed deep learning
• Fanout-Cache
• GPU utilization