An Adaptive Heterogeneous Architecture for High-Ratio, High-Throughput Lossless Compression

2026-09-28

AI Summary

This study addresses the inherent conflict in lossless compression between industrial-grade high throughput and the superior compression ratios achieved by context-mixing algorithms. To reconcile this trade-off, we propose GPX, an adaptive heterogeneous architecture that integrates GPU-based invertible domain transforms, AVX2-accelerated multi-hypothesis sequence optimization, and a sub-15-microsecond structural decision probe to enable efficient compression of structured data streams. This approach establishes a Pareto frontier advantage under RFC 8878-compatible configurations. Evaluated on the Silesia benchmark, GPX achieves a compressed size of 54.46 MiB and a throughput of 191.90 MiB/s, strictly dominating the official Zstandard levels 10–14 while demonstrating zero-tuning generalization capability.

Abstract

Modern lossless data compression enforces an acute dichotomy: industrial streaming codecs (e.g., Zstandard) prioritize throughput (10 to 1000 MiB/s) at the expense of compression density, while context-mixing algorithms achieve superior ratios at unusable serial speeds (0.1 to 1 MiB/s). We present GPX, an adaptive heterogeneous compression architecture for structured data streams. While optimized for high-throughput enterprise and scientific large-block workloads (>= 4 MiB), GPX operates transparently across arbitrary stream lengths down to sub-kilobyte inputs (N >= 256 B). GPX combines a sub-15-microsecond native structural decision probe, GPU-native reversible domain transforms, AVX2 multi-hypothesis sequence optimization, and adaptive entropy boundaries. On the canonical Silesia benchmark (202.12 MiB) under a preregistered 7-repeat Median+IQR protocol (W = 6 workers), GPX Track A produces standard RFC 8878-compliant .zst streams that compress to 56.27 MiB at 235.41 MiB/s with native line-speed decompression (3074.27 MiB/s on unmodified libzstd), establishing non-domination among evaluated RFC 8878-compatible configurations. GPX Track B achieves 54.49 MiB at 259.76 MiB/s (CI [250.7, 287.6]), establishing an empirical Pareto frontier point and dominating Stock Levels 8-14. GPX Track A+B achieves 54.46 MiB at 191.90 MiB/s (CI [177.8, 206.7]), strictly dominating official Stock Zstandard Levels 10-14 under non-overlapping 95% Bootstrap confidence intervals (182.18 KB smaller than Level 14 and 1.15x to 18.70x faster than Levels 10-14). Zero-tuning out-of-sample generalization is confirmed across Canterbury (-0.650%), Calgary (-0.124%), and a 102.45 MB modern compiled binary (-0.246%), while enwik8 (+0.084%) documents the authentic failure mode of splitter over-segmentation. All outputs are 100% SHA-256 bit-exact round-trip verified.

Problem

• Lossless Compression
• Compression Ratio
• Throughput
• Context Mixing
• Data Compression

Innovation

• Adaptive Heterogeneous Architecture
• Lossless Compression
• GPU-native Reversible Transforms
• AVX2 Multi-hypothesis Optimization
• Pareto Frontier