Adaptive Parallel Downloader for Large Genomic Datasets

2025-08-07

AI Summary

To address low bandwidth utilization and poor download efficiency caused by static concurrency strategies in TB-scale data retrieval from public genomic repositories (e.g., SRA/ENA), this paper proposes the first adaptive parallel download framework tailored for large-scale biological data. We formulate the download process as an online optimization problem and design a utility-driven gradient descent algorithm that dynamically adjusts HTTP/FTP concurrent connection counts in real time, automatically balancing throughput gains against system overhead. Integrating online learning with I/O-level optimizations, our client-side scheduler achieves lightweight yet high-efficiency resource management. Evaluated on real-world genomic datasets, our framework achieves up to 4.0× speedup over state-of-the-art tools (e.g., fasterq-dump, Aspera), with an average 2.1× acceleration under high-bandwidth conditions—significantly enhancing the efficiency of public biological data acquisition.

Technology Category

• Search and Optimization: Distributed Search
• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Machine Learning: Online Learning & Bandits

Application Category

• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets

Abstract

Modern next-generation sequencing (NGS) projects routinely generate terabytes of data, which researchers commonly download from public repositories such as SRA or ENA. Existing download tools often employ static concurrency settings, leading to inefficient bandwidth utilization and prolonged download times due to their inability to adapt to dynamic network conditions. We introduce FastBioDL, a parallel file downloader designed for large biological datasets, featuring an adaptive concurrency controller. FastBioDL frames the download process as an online optimization problem, utilizing a utility function and gradient descent to adjust the number of concurrent socket streams in real-time dynamically. This approach maximizes download throughput while minimizing resource overhead. Comprehensive evaluations on public genomic datasets demonstrate that FastBioDL achieves up to $4x$ speedup over state-of-the-art tools. Moreover, in high-speed network experiments, its adaptive design was up to $2.1x$ faster than existing tools. By intelligently optimizing standard HTTP or FTP downloads on the client side, FastBioDL provides a robust and efficient solution for large-scale genomic data acquisition, democratizing high-performance data retrieval for researchers without requiring specialized commercial software or protocols.

Problem

• Inefficient bandwidth use in genomic data downloads
• Static concurrency settings prolong download times
• Need adaptive tool for large biological datasets

Innovation

• Adaptive concurrency control for dynamic networks
• Online optimization with utility function
• Real-time gradient descent adjustment