Open Science Data Federation -- operation and monitoring

2026-05-14

AI Summary

This work addresses the challenge of fragmented infrastructure hindering efficient distribution and sharing of large-scale scientific data. Building upon StashCache, the authors construct a global research data access network by expanding data origins and cache nodes and enabling multi-protocol access. They introduce a federated architecture that uniquely integrates access control, real-time monitoring, and usage metering to comply with NSF data-sharing policies. The resulting system has been scaled into a national research data federation and now serves as a critical component of the U.S. national cyberinfrastructure, effectively supporting multidisciplinary teams in accessing data globally with high efficiency and precise usage accounting.

Abstract

Extensive data processing is becoming commonplace in many fields of science. Distributing data to processing sites and providing methods to share the data with collaborators efficiently has become essential. The Open Science Data Federation (OSDF) builds upon the successful StashCache project to create a global data access network. The OSDF expands the StashCache project to add new data origins and caches, access methods, monitoring, and accounting mechanisms. Additionally, the OSDF has become an integral part of the U.S. national cyberinfrastructure landscape due to the sharing requirements of recent NSF solicitations, which the OSDF is uniquely positioned to enable. The OSDF continues to be utilized by many research collaborations and individual users, which pull the data to many research infrastructures and projects.

Problem

• data distribution
• data sharing
• scientific data processing
• cyberinfrastructure
• research collaboration

Innovation

• Open Science Data Federation
• data federation
• distributed caching
• research data sharing
• cyberinfrastructure