Lossless Compression Performance for PETRA III Datasets

2026-07-31

AI Summary

This study addresses the tension between escalating data volumes and sustainable archiving in large-scale scientific facilities, necessitating efficient lossless compression solutions. The authors conduct a systematic evaluation of 212 TiB of data from ten experimental categories at the PETRA III synchrotron radiation source, revealing for the first time the pronounced heterogeneity in compressibility across data types. They benchmark nine state-of-the-art compression algorithms—including Zstandard, LZ4, and ZPAQ—across these categories. Leveraging Pareto front analysis and data classification, they propose a heterogeneous strategy that adaptively selects compressors based on file type. This approach achieves significantly superior overall performance compared to uniform strategies, yielding 1.6–2.1× higher compression ratios while maintaining throughputs ranging from 2 to 900 MiB/s.

Abstract

Large-scale research facilities increasingly face the challenge of managing rapidly growing data volumes while maintaining sustainable archival infrastructures. We present the first comprehensive study of data heterogeneity and lossless general-purpose compression performance for representative datasets from the PETRA III synchrotron radiation source. Our corpus comprises more than 212 TiB of raw and processed data from ten experiments spanning multiple beamlines, detector systems, and scientific workflows. We observe substantial heterogeneity both between and within experiments, resulting in compression ratios that vary by more than two orders of magnitude across datasets. Evaluating nine widely used lossless compression tools, we find that Zstandard and LZ4 consistently occupy the high-throughput region of the Pareto front, whereas ZPAQ achieves the highest compression ratios. Furthermore, heterogeneous compression strategies that adapt compressor choice to the underlying file category outperform uniform compression policies. Extrapolating from the benchmarked datasets to the full PETRA III non-tape storage system, we estimate achievable compression ratios ranging from approximately 1.6 at $\sim$900 MiB/s throughput to 2.1 at $\sim$2 MiB/s. These results provide a quantitative basis for future archival and storage decisions at PETRA III, its future successor, PETRA IV, and other large-scale scientific facilities.

Problem

• lossless compression
• data heterogeneity
• scientific data management
• archival storage
• PETRA III

Innovation

• lossless compression
• data heterogeneity
• adaptive compression strategy
• scientific data management
• Pareto-optimal compression