TROOP: At-the-Roofline Performance for Vector Processors on Low Operational Intensity Workloads

2025-08-05

AI Summary

To address the low L1 memory bandwidth utilization and sub-roofline performance of vector processors on low-compute-intensity kernels (e.g., GEMV, DOTP), this work proposes microarchitectural optimizations targeting memory-intensive workloads: decoupling load-store interfaces, enhancing vector chaining, introducing shadow buffers to hide VRF access conflicts, and applying address perturbation to improve L1 parallel memory access efficiency. To our knowledge, this is the first implementation achieving roofline-bound performance for low-data-reuse scenarios on an open-source vector processor fabricated in 12 nm. Experimental results show speedups of 1.5×, 2.2×, and 2.6× for GEMV, DOTP, and AXPY, respectively, with up to 45% energy-efficiency improvement. DOTP achieves 38 DP-GFLOPs/W, while GEMM sustains 61 DP-GFLOPs/W—both under a silicon area overhead of less than 7%.

Technology Category

• Machine Learning: Hardware-aware ML
• Computer Vision: Learning & Optimization for CV
• Search and Optimization: Distributed Search

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Web performance, measurement, and characterization
• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

The fast evolution of Machine Learning (ML) models requires flexible and efficient hardware solutions as hardwired accelerators face rapid obsolescence. Vector processors are fully programmable and achieve high energy efficiencies by exploiting data parallelism, amortizing instruction fetch and decoding costs. Hence, a promising design choice is to build accelerators based on shared L1-memory clusters of streamlined Vector Processing Elements (VPEs). However, current state-of-the-art VPEs are limited in L1 memory bandwidth and achieve high efficiency only for computational kernels with high data reuse in the Vector Register File (VRF), such as General Matrix Multiplication (GEMM). Performance is suboptimal for workloads with lower data reuse like General Matrix-Vector Multiplication (GEMV). To fully exploit available bandwidth at the L1 memory interface, the VPE micro-architecture must be optimized to achieve near-ideal utilization, i.e., to be as close as possible to the L1 memory roofline (at-the-roofline). In this work, we propose TROOP, a set of hardware optimizations that include decoupled load-store interfaces, improved vector chaining, shadow buffers to hide VRF conflicts, and address scrambling techniques to achieve at-the-roofline performance for VPEs without compromising their area and energy efficiency. We implement TROOP on an open-source streamlined vector processor in a 12nm FinFET technology. TROOP achieves significant speedups of 1.5x, 2.2x, and 2.6x, respectively, for key memory-intensive kernels such as GEMV, DOTP and AXPY, achieving at-the-roofline performance. Additionally, TROOP enhances the energy efficiency by up to 45%, reaching 38 DP-GFLOPs/W (1 GHz, TT, 0.8V) for DOTP while maintaining a high energy efficiency of 61 DP-GFLOPs/W for GEMMs, incurring only a minor area overhead of less than 7%.

Problem

• Optimizing vector processors for low operational intensity workloads
• Enhancing L1 memory bandwidth utilization in VPEs
• Improving performance and energy efficiency without significant area overhead

Innovation

• Decoupled load-store interfaces for bandwidth optimization
• Improved vector chaining to enhance performance
• Shadow buffers to hide VRF conflicts efficiently