Problem
Research questions and friction points this paper is trying to address.
data distribution
scientific data sharing
global data access
big data processing
cyber-infrastructure
Using the Open Science Data Federation for data distribution: Big Bear Solar Observatory use case
🤖 AI Summary
This work addresses the challenge of efficient scientific data distribution and global sharing, particularly in response to the National Science Foundation’s mandate for open data access. To meet this requirement, the project integrates data from the Big Bear Solar Observatory (BBSO) into the Open Science Data Federation (OSDF). By extending the StashCache architecture with multi-source data integration, distributed caching nodes, a unified access interface, novel access mechanisms, and monitoring and accounting capabilities, the solution enables globally localized caching of BBSO data. This approach significantly enhances data availability and access efficiency while supporting the deployment of distributed processing pipelines for full-disk solar imagery at any location worldwide. Consequently, OSDF is positioned as a critical component of the U.S. national research infrastructure.
📝 Abstract
The growing demand for extensive data processing is now a standard in many scientific fields. Efficiently distributing data to processing sites and enabling seamless sharing has become crucial. The Open Science Data Federation (OSDF) builds on the success of the StashCache project to establish a global data distribution network. By expanding StashCache, OSDF integrates additional data origins and caches, enhancing accessibility and performance (20 origins and 30 caches), new access methods, and monitoring and accounting mechanisms. Additionally, the OSDF has become essential to the US national cyber-infrastructure landscape due to the sharing requirements of recent NSF solicitations. One use case for the OSDF is the data access to the Big Bear Solar Observatory (BBSO). Integrating the BBSO data into the OSDF provided standard and reliable data access. Moreover, the OSDF caches provide local data worldwide. Using the OSDF and the BBSO data, creating a pipeline to apply image processing techniques to all images from BBSO anywhere on the planet was possible.
Innovation
Methods, ideas, or system contributions that make the work stand out.
Open Science Data Federation
StashCache
data distribution
Big Bear Solar Observatory
distributed caching
🔎 Similar Papers
No similar papers found.
💼 Related Jobs
