matrix blocking

Existing ARM linear algebra libraries fail to fully exploit the Scalable Matrix Extension (SME) architecture, particularly suffering from severe performance bottlenecks in large-scale General Matrix Multiplication (GEMM). To address this, we propose MpGEMM—the first open-source, high-performance GEMM library specifically designed for ARM SME. We conduct the first systematic microarchitectural characterization of SME and derive three key optimization principles: cache-aware blocking, dynamic on-the-fly transpose-and-pack, and fully register-resident tile-based microkernels. Leveraging multi-vector load instructions and fine-grained tile register scheduling, MpGEMM achieves a 1.23× speedup over Apple’s Accelerate framework on the Apple M4 Pro—outperforming leading open-source libraries. We further validate its effectiveness on realistic large-model workloads, including DeepSeek and LLaMA, demonstrating substantial end-to-end inference acceleration.

This work addresses the challenge of dynamically determining the optimal thread count for General Matrix Multiplication (GEMM) on multi-core shared-memory systems. To this end, the authors propose the Architecture- and Data Structure-Aware Linear Algebra library (ADSALA), which integrates a runtime machine learning model into a BLAS library for the first time. By jointly considering task characteristics, hardware context, and conventional optimizations such as blocking, ADSALA dynamically selects the best thread configuration. Evaluated on dual-socket HPC nodes featuring Intel Cascade Lake and AMD Zen 3 architectures, the approach achieves a 25%–40% speedup over traditional BLAS implementations for GEMM workloads with memory footprints under 100 MB, significantly enhancing performance adaptability.

Deploying GEMM on tile-based multi-PE accelerators faces challenges of deployment complexity and deep hardware-software coupling. To address this, we propose an end-to-end automated deployment framework. Our approach introduces the novel “Design in Tiles” paradigm, integrating configurable execution modeling, hardware-aware automatic mapping, hierarchical tiling scheduling, and compute-memory co-optimized compilation. For the first time, we achieve superior PE utilization over NVIDIA GH200’s expert-tuned library on a large-scale 32×32 tile configuration. At FP8 precision, our framework delivers 1979 TFLOPS peak performance and accelerates diverse matrix shapes by 1.2–2.0× relative to GH200. This work bridges the compilation gap between configurable hardware architectures and high-level computational graphs, establishing a general, efficient, and scalable methodology for automatic mapping onto domain-specific accelerators.

This work addresses the redundant memory overhead in existing BLAS APIs caused by mandatory data packing and unpacking during serialized GEMM operations. The authors propose a BLAS-compatible decomposition method for GEMM kernels that, for the first time, enables propagation of data layouts across consecutive GEMM calls, thereby eliminating repeated intermediate data transformations while preserving semantic correctness. Implemented in C++ and integrated into a pure BLAS calling chain, the approach is evaluated on both x86 (AVX-512) and RISC-V (RVV 1.0) architectures. Experimental results demonstrate an average 2.25× speedup over OpenBLAS on Intel x86 platforms, achieving performance comparable to Intel MKL, with practical efficacy further validated in Llama-3.2 inference workloads.

High-precision matrix multiplication (e.g., FP64×2) remains challenging to implement efficiently on existing hardware due to the absence of native support for extended-precision arithmetic. Method: This paper proposes a novel approximation approach based on cascading low-precision operators: using standard FP64 GEMM as the fundamental building block, it designs a ten-stage cascade—integrated with error modeling and precision decomposition-recomposition—to reconstruct Goto’s algorithm within the BLIS framework. Contribution/Results: To our knowledge, this is the first method enabling FP64×2-level numerical accuracy using only native FP64 hardware primitives, establishing a new paradigm of “constructing high-precision linear algebra from low-precision primitives.” Experiments demonstrate that the approach achieves near-FP64 GEMM throughput while significantly improving numerical accuracy, thereby validating the feasibility of co-optimizing precision and performance.

This work addresses the conflict between conventional page-granularity interleaved memory layouts and the locality demands of GEMM operations on chiplet-based GPUs, which incurs substantial remote HBM access overhead. To resolve this, the authors propose Chiplet-Contiguous Layout—a hardware- and OS-agnostic global memory organization that colocates each chiplet’s local data contiguously, thereby enabling locality-aware scheduling while remaining compatible with standard 4KB page management. This approach is the first to unify data locality optimization for GEMM with page-granular memory management in both large language model inference and training. Experiments demonstrate that, compared to 4KB interleaved layouts, the proposed method reduces remote HBM traffic by 24.7× and 19.2× on Qwen-3 30B and Llama-3.1 70B, respectively, and further achieves 4.1× and 2.1× reductions over coarse-grained locality-aware placements.

This work addresses the limitations of existing sparse general matrix-matrix multiplication (SpGEMM) accelerators, which rely on static dataflows that struggle to simultaneously achieve high data reuse and load balance. To overcome these inherent constraints, we propose SegFold, the first SpGEMM accelerator architecture incorporating fine-grained dynamic scheduling and dynamic remapping of partially completed work into its dataflow design. SegFold employs a customized memory controller to exploit data reuse within local windows and integrates dynamic task distribution with a reconfigurable merging network to ensure balanced workload allocation across processing units. Experimental results demonstrate that SegFold achieves an average speedup of 1.95× over state-of-the-art SpGEMM accelerators and up to 5.3× improvement compared to the best static configuration, across a wide range of sparsity patterns and matrix sizes.

In multi-chiplet GPUs, GEMM operations suffer significant performance and energy efficiency degradation due to remote HBM accesses, while determining optimal data layouts and CTA scheduling strategies remains challenging. This work proposes a fast, functional-level, tile-granularity locality simulator that models CTA scheduling, per-chiplet L2 caches, and local/remote HBM accesses to efficiently evaluate inter-chiplet traffic for full-scale GEMMs in large language models. The study reveals, for the first time, that CTA traversal order is a first-order design variable governing remote traffic. Through AI-driven design space exploration, it demonstrates that a 2D block-interleaved traversal strategy can reduce remote traffic by up to 5.1× compared to the best 1D strategy, with traffic differences as high as 90× across different scheduling policies under identical GEMM configurations.

This work proposes a tuning-free General Matrix Multiplication (GEMM) approach based on generalized Hilbert space-filling curves, addressing the performance instability of conventional GEMM across diverse hardware architectures and matrix shapes. By leveraging high-locality computation partitioning, cache-oblivious blocking, and optimized tensor layouts, the method achieves consistently high performance independent of both platform and operand dimensions. It further integrates communication-avoiding strategies to minimize data movement, ensuring communication optimality. The implementation is remarkably concise and demonstrates superior performance over leading vendor libraries across a wide range of CPU platforms and GEMM problem shapes, achieving up to a 2× geometric mean speedup while maintaining both portability and communication efficiency.

This work proposes FalconGEMM, a cross-platform framework designed to overcome hardware peak performance limitations and enhance the efficiency of large language model training and inference by enabling the first practical deployment of low-complexity matrix multiplication algorithms (LCMAs) on heterogeneous hardware. The framework integrates three core components: cross-platform code generation supporting GPUs (H20, A100) and CPUs (ARM, x86), execution optimization leveraging group-level parallelism and on-chip data reuse, and a lightweight performance model guiding optimal strategy selection. Experimental results demonstrate that FalconGEMM outperforms state-of-the-art GEMM libraries by 7.59%–17.85% across diverse hardware platforms and data types, and achieves speedups of 12.41%–55.61% over existing LCMA approaches such as AlphaTensor, thereby bridging the gap between theoretical algorithms and production-grade applications.