python performance optimization

To address the challenges of cross-platform orchestration and fragmented resource scheduling in hybrid HPC–ML workflows, this paper proposes a service-oriented, scalable runtime architecture. Building upon the RADICAL-Pilot framework, we introduce the first service-oriented execution model enabling dynamic, multi-granularity, low-overhead coordination of heterogeneous HPC and ML tasks. Our approach unifies resource abstraction across platforms, implements distributed task scheduling, and jointly orchestrates AI and HPC workloads—thereby enabling seamless coupling and coordinated scheduling between on-premises exascale supercomputers and cloud environments. Experimental evaluation on an exascale prototype system demonstrates concurrent deployment of multiple ML models with runtime overhead under 2%. The architecture successfully supports three representative data-driven scientific applications, effectively overcoming the traditional siloing of HPC and ML workflows.

Traditional space-filling experimental designs—e.g., Latin Hypercube Sampling (LHS)—fail for complex simulation models featuring optional and nested hierarchical inputs. To address this, we propose *raxpy*, the first open-source Python toolkit supporting optional and hierarchical inputs. Its core comprises an automatic parsing mechanism leveraging Python type annotations and runtime introspection, tightly integrated with an enhanced hierarchical Latin hypercube sampling algorithm. This enables generation of experimental designs with high input-space coverage and sample uniformity, while natively supporting multiprocessing and distributed execution. Numerical experiments demonstrate that *raxpy* significantly outperforms baseline methods in both coverage and uniformity (p < 0.01). Released under the MIT license, *raxpy* has been successfully integrated into multiple industrial-scale simulation workflows.

This work addresses key engineering challenges in the systematic evolution of large language models (LLMs)—including cache reuse, context management, agent scheduling, and access control—stemming from the absence of a unified architectural framework. To bridge this gap, the paper proposes the six-layer Intelligent Computing Architecture Model (ICAM), which unifies LLM-native system design through a dual-plane view comprising a probabilistic execution plane and a deterministic control plane. Grounded in three design principles—semantic locality, context budgeting, and agent acceleration—ICAM establishes interface contracts and design axioms for model-native systems. By integrating analogical analysis, architectural abstraction, and system modeling, the framework synthesizes advances in LLM-as-OS, memory management, multi-agent coordination, and security governance. System-level empirical validation confirms the efficacy of the proposed laws, elucidates parallels and divergences between model-native computing and traditional computer architecture, and outlines promising directions for future research.

This work addresses the high end-to-end latency in composite AI applications caused by frequent invocations of external machine learning models, a challenge poorly mitigated by conventional Python optimization techniques due to language dynamism, dynamic dispatch, and abrupt variable mutations. To overcome these limitations, the authors propose an automatic parallelization framework that synergistically combines static compilation with runtime coordination. The framework supports a highly expressive subset of Python while preserving semantic correctness and requires minimal manual intervention to uncover parallelism inherent in the program. It is particularly effective in scenarios involving heavyweight external components, achieving up to 6.4× end-to-end speedup on real-world composite AI applications and substantially reducing execution latency.

Programming heterogeneous exascale HPC systems is hindered by the proliferation of complex, incompatible programming models (e.g., CUDA, SYCL, OpenMP) and the lack of traceability across CPU/GPU execution contexts. Method: We propose the first semantic-aware, full-stack API tracing framework built upon LTTng kernel tracing, integrating user-space dynamic instrumentation with multi-model API signature parsing to capture fine-grained, low-overhead, configurable API call chains across hardware and programming abstractions. Contribution/Results: Unlike conventional tracers logging only function names and timestamps, our framework enables cross-vendor, cross-abstraction behavioral correlation and end-to-end call-chain reconstruction in real HPC applications. It accurately identifies cross-model performance bottlenecks and implementation flaws, improving debugging efficiency by over 3× and significantly enhancing portability and debuggability of heterogeneous programming models.

This work addresses the orchestration bottlenecks faced by ultra-large-scale Sim-AI workflows on leadership-class supercomputers, which arise from task heterogeneity and extreme ensemble sizes. To overcome these challenges, the authors propose EnsembleLauncher, a recursively hierarchical and fully decentralized workflow orchestrator that introduces a decentralized control plane and a programmable scheduling policy interface, thereby surpassing conventional tools in both scalability and scheduling flexibility. Experiments on the Aurora supercomputer demonstrate that EnsembleLauncher can efficiently schedule system-wide resources to support up to 8 million serial tasks, achieving more than a fourfold performance improvement over state-of-the-art alternatives. Furthermore, it significantly enhances resource utilization for workloads with high task variance and active learning pipelines.

This work addresses the significant degradation in inference throughput caused by GPU memory constraints when concurrently deploying multiple large language models on shared heterogeneous hardware, where resource scheduling, model offloading, and preemption become critical bottlenecks. Through empirical methodologies—including cross-platform performance profiling, layer-wise offloading experiments, and fine-grained decomposition of preemption overhead—the study systematically uncovers, for the first time, the nonlinear relationship between offloading and throughput decline. It further identifies model state reloading as the primary source of preemption overhead. The findings reveal that smaller models are more sensitive to reduced GPU residency, and that such overhead is jointly influenced by model architecture and hardware characteristics. These insights motivate a scheduler design that integrates model-specific sensitivity with data migration costs, offering crucial guidance for building efficient multi-model serving systems.

Existing cluster-level full-stack simulation struggles to simultaneously achieve high fidelity and high performance. This work proposes the concept of a “simulation-native operating system,” which integrates simulation control and orchestration into the OS kernel, thereby constructing a full-stack simulation framework built upon the Linux virtualization stack. The framework employs four key mechanisms—simulation-oriented scheduling, real-time memory hierarchy management, simulation-aware inter-process communication (IPC), and distributed simulation orchestration—to seamlessly co-execute real and simulated components without requiring modifications to production systems. Experimental results demonstrate that this approach significantly enhances the performance and configuration exploration efficiency of large-scale cluster simulations while preserving full-stack fidelity.

HPC workloads are becoming increasingly heterogeneous, rendering traditional static heuristic schedulers inadequate for dynamic resource demands. To address this, we propose SchedTwin—the first real-time digital twin system for HPC job scheduling. It continuously ingests runtime event streams to drive high-fidelity discrete-event simulation, enabling rapid online evaluation of “what-if” scenarios across multiple scheduling policies and facilitating goal-driven, closed-loop adaptive scheduling. Deeply integrated with the PBS scheduler, SchedTwin achieves low-overhead (sub-10-second decision latency) and high-accuracy online policy optimization. Experimental evaluation in production environments demonstrates that SchedTwin significantly outperforms mainstream static schedulers—overcoming the longstanding dual bottlenecks of adaptability and timeliness inherent in conventional HPC scheduling approaches.

This work addresses the lack of a universal, flexible, and cluster-agnostic workload representation in existing distributed machine learning systems, which hinders efficient design space exploration. To overcome this limitation, the paper introduces Flint, a novel framework that leverages the intermediate representation of machine learning compilers to extract workload graphs for clusters of arbitrary scale—without requiring actual hardware execution. By decoupling workload modeling from underlying hardware specifics and validating accuracy through execution traces, Flint ensures both fidelity and portability. Experimental results demonstrate that Flint effectively enables flexible and efficient design space exploration while substantially reducing evaluation overhead.