A Scalable In Transit Solution for Comprehensive Exploration of Simulation Data

2025-10-19

AI Summary

In high-performance computing, simulation data volumes vastly exceed storage capacity, while conventional in-situ analysis relies on prior knowledge—such as predefined visualization parameters and static resource allocation—limiting adaptability and scalability. Method: This paper proposes an in-transit analysis service for 3D simulation data, integrating a lightweight decentralized architecture, dynamic elastic resource scheduling, and an efficient lossy compression technique tailored to 3D data. The service enables concurrent multi-task analysis without MPI synchronization. Contribution/Results: Its core innovation lies in eliminating dependence on a priori parameters and fixed resources, enabling on-demand offloading of analytical workloads onto shared infrastructure. Experiments demonstrate substantial reductions in storage overhead, enhanced analytical flexibility, and improved utilization of computational resources—thereby supporting dynamic, real-time exploration of large-scale simulations.

Technology Category

• Data Mining & Knowledge Management: Scalability, Parallel & Distributed Systems
• Search and Optimization: Sampling/Simulation-based Search
• Computer Vision: 3D Computer Vision

Application Category

• Web Mining and Content Analysis: Web data generation and simulation
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Systems and Infrastructure for Web, Mobile and WoT: Data management and stream processing for Web, mobile and wireless applications

Abstract

As simulations produce more data than available disk space on supercomputers, many simulations are employing in situ analysis and visualization to reduce the amount of data that needs to be stored. While in situ visualization offers potential for substantial data reduction, its efficacy is hindered by the need for a priori knowledge. First, we need to know what visualization parameters to use to highlight features of interest. Second, we do not know ahead of time how much resources will be needed to run the in situ workflows, e.g. how many compute nodes will be needed for in situ work. In this work, we present SeerX, a lightweight, scalable in-transit in situ service that supports dynamic resource allocation and lossy compression of 3D simulation data. SeerX enables multiple simulations to offload analysis to a shared, elastic service infrastructure without MPI synchronization.

Problem

• Reducing simulation data storage through in situ visualization
• Overcoming a priori knowledge limitations in visualization parameters
• Addressing dynamic resource allocation for in situ workflows

Innovation

• Lightweight scalable in-transit service for simulations
• Dynamic resource allocation for in-situ workflows
• Lossy compression enabling shared elastic service infrastructure