Benchmarking the Open Science Data Federation services to develop XRootD best practices

2026-05-13

AI Summary

This study addresses the lack of systematic evaluation of XRootD performance across diverse configurations and network conditions, which has hindered the development of best practices for scientific data distribution. We present the first multidimensional benchmarking framework deployed on the National Research Platform (NRP), comprehensively assessing XRootD and Pelican under varying file sizes, numbers of parallel streams, client distances, transfer tools, storage types, buffer settings, and bandwidth constraints. The evaluation integrates multiple clients—including wget, curl, and pelican—as well as HTCondor’s native transfer mechanisms for comparative analysis. Our findings yield a set of actionable recommendations for optimizing XRootD configurations, providing empirical guidance for the deployment and tuning of the Open Science Data Federation (OSDF) global data distribution network.

Abstract

Research has become dependent on processing power and storage, one crucial aspect being data sharing. The Open Science Data Federation (OSDF) project aims to create a scientific global data distribution network based on the Pelican Platform. OSDF relies on the XRootD and Pelican projects. Nevertheless, OSDF must understand the XRootD limits under various configuration options, including transfer rate limits, proper buffer configuration, and storage type effect. We have thus executed a set of benchmarks to create a set of recommendations to share with the XRootD and Pelican teams. This work describes the tests and results performed using National Research Platform (NRP) hosts. The tests cover various file sizes and parallel streams and use clients from various distances from the server host. We also used several standalone clients (wget, curl, pelican) and the native HTCondor file transfer mechanisms. Applying the methodology creates a possibility to track how XRootD and the Pelican layer perform in different scenarios.

Problem

• XRootD
• Open Science Data Federation
• data distribution
• performance benchmarking
• Pelican Platform

Innovation

• XRootD
• benchmarking
• Open Science Data Federation
• Pelican Platform
• data transfer optimization