Large Data Acquisition and Analytics at Synchrotron Radiation Facilities

2026-02-05

AI Summary

This work addresses the challenges posed by terabyte-scale data generated at synchrotron facilities under limited beamtime, including difficulties in remote monitoring, complex data management, and the absence of real-time quality assessment. To overcome these issues, we designed and deployed a secure server-based, real-time networked framework that, for the first time, enables unified remote monitoring and data quality evaluation across multiple beamlines. Built on web technologies, the system integrates real-time data acquisition, visualization, file management, and user dashboards. It has successfully managed 50–100 TB of data and over 10 million files across three beamlines, with validation from 43 research groups demonstrating significantly improved data accessibility, optimized experimental workflows, and reduced operational overhead.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Search and Optimization: Distributed Search
• Planning, Routing, and Scheduling: Optimization of Spatio-temporal Systems

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications
• Security and Privacy: Data transparency and provenance
• Web Mining and Content Analysis: Web data quality in the era of algorithmically-generated content

Abstract

Synchrotron facilities like the Cornell High Energy Synchrotron Source (CHESS) generate massive data volumes from complex beamline experiments, but face challenges such as limited access time, the need for on-site experiment monitoring, and managing terabytes of data per user group. We present the design, deployment, and evaluation of a framework that addresses CHESS's data acquisition and management issues. Deployed on a secure CHESS server, our system provides real time, web-based tools for remote experiment monitoring and data quality assessment, improving operational efficiency. Implemented across three beamlines (ID3A, ID3B, ID4B), the framework managed 50-100 TB of data and over 10 million files in late 2024. Testing with 43 research groups and 86 dashboards showed reduced overhead, improved accessibility, and streamlined data workflows. Our paper highlights the development, deployment, and evaluation of our framework and its transformative impact on synchrotron data acquisition.

Problem

• Synchrotron Radiation
• Large Data Acquisition
• Data Management
• Beamline Experiments
• Real-time Monitoring

Innovation

• real-time remote monitoring
• synchrotron data management
• web-based analytics
• high-throughput data acquisition
• beamline workflow optimization