Evaluating the Energy Efficiency of NPU-Accelerated Machine Learning Inference on Embedded Microcontrollers

2025-09-22

AI Summary

Addressing stringent energy efficiency, latency, and memory constraints for machine learning inference on battery-powered edge devices, this work proposes a hardware acceleration framework leveraging an embedded Neural Processing Unit (NPU). We employ the ARM Cortex-M55 + Ethos-U55 heterogeneous architecture and introduce a novel GPIO-triggered high-precision power measurement methodology to isolate and quantify the net energy consumption of NPU inference by accurately subtracting idle power. Compared to CPU-only execution, NPU acceleration achieves 7×–125× latency reduction and up to 143× lower net energy per inference. Moreover, we demonstrate, for the first time on microcontrollers, successful deployment of complex quantized models such as SSD-MobileNet. Our core contributions are: (1) establishing a rigorous, reproducible benchmarking methodology for embedded NPU energy efficiency; and (2) empirically validating simultaneous breakthroughs in functional scalability—supporting larger models—and superior energy efficiency.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Natural Language Processing: Learning & Optimization for NLP
• Computer Vision: Adversarial Attacks & Robustness

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Search and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

The deployment of machine learning (ML) models on microcontrollers (MCUs) is constrained by strict energy, latency, and memory requirements, particularly in battery-operated and real-time edge devices. While software-level optimizations such as quantization and pruning reduce model size and computation, hardware acceleration has emerged as a decisive enabler for efficient embedded inference. This paper evaluates the impact of Neural Processing Units (NPUs) on MCU-based ML execution, using the ARM Cortex-M55 core combined with the Ethos-U55 NPU on the Alif Semiconductor Ensemble E7 development board as a representative platform. A rigorous measurement methodology was employed, incorporating per-inference net energy accounting via GPIO-triggered high-resolution digital multimeter synchronization and idle-state subtraction, ensuring accurate attribution of energy costs. Experimental results across six representative ML models -including MiniResNet, MobileNetV2, FD-MobileNet, MNIST, TinyYolo, and SSD-MobileNet- demonstrate substantial efficiency gains when inference is offloaded to the NPU. For moderate to large networks, latency improvements ranged from 7x to over 125x, with per-inference net energy reductions up to 143x. Notably, the NPU enabled execution of models unsupported on CPU-only paths, such as SSD-MobileNet, highlighting its functional as well as efficiency advantages. These findings establish NPUs as a cornerstone of energy-aware embedded AI, enabling real-time, power-constrained ML inference at the MCU level.

Problem

• Evaluating NPU impact on energy efficiency for ML inference on microcontrollers
• Addressing strict energy and latency constraints for ML on battery-operated edge devices
• Measuring NPU vs CPU performance gains for embedded machine learning models

Innovation

• Using NPU hardware acceleration for MCU inference
• Employing GPIO-triggered energy measurement methodology
• Achieving significant latency and energy improvements