Improving compiler support for SIMD offload using Arm Streaming SVE

2025-06-02

AI Summary

Arm’s core-bypass SIMD accelerators (e.g., Scalable Matrix Extension, SME) lack compiler-supported automatic offloading, hindering efficient utilization of their streaming capabilities. Method: This paper presents the first systematic reconstruction of vectorization decision-making for core-bypass accelerators within LLVM/MLIR. It introduces a precise cost model tailored to SVE Streaming Mode (SSVE), formal semantic modeling of SSVE instructions, and reproducible offloading heuristics—enabling end-to-end, selective C/C++ loop offloading to SME. Offloading is triggered only when performance gains are significant, avoiding speculative optimization. Results: Evaluated on representative matrix workloads, the approach achieves 2.3× average speedup and >91% offloading decision accuracy. Its core contribution is the first compiler-level automation framework for SSVE-based streaming offloading, breaking the traditional constraint that vectorization targets only on-die vector units.

Technology Category

• Machine Learning: Hardware-aware ML
• Search and Optimization: Sampling/Simulation-based Search
• Humans and AI: Human-in-the-loop Machine Learning

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

The wider adoption of tightly coupled core-adjacent accelerators, such as Arm Scalable Matrix Extension (SME), hinges on lowering software programming complexity. In this paper, we focus on enabling the use of SME architecture in Streaming Scalable Vector Extension (SSVE) mode for workloads written in C/C++. While current compilers optimize loops for all types of SIMD instructions, these techniques primarily target vector units within the core and falter when applied to disaggregated, core-adjacent SIMD accelerators. Our goal is to enable the compiler to automatically generate code for such accelerators only when profitable. To this end, we investigate a path towards performant, precise, and repeatable computation offloading through two compiler ecosystems. We revisit LLVM compiler passes, MLIR transforms and their associated cost models, and heuristics. We hope that these insights can provide directions for evolving compiler capabilities towards automatic code generation for this next-generation vector processing paradigm.

Problem

• Enabling SIMD offload for Arm SME in C/C++ workloads
• Optimizing compiler support for core-adjacent SIMD accelerators
• Automatic code generation for profitable SIMD acceleration

Innovation

• Enabling SME architecture in SSVE mode
• Revisiting LLVM passes and MLIR transforms
• Automatic code generation for SIMD accelerators