EdgeMark: An Automation and Benchmarking System for Embedded Artificial Intelligence Tools

2025-02-03

AI Summary

Selecting appropriate tools for embedded AI (eAI/TinyML) development remains challenging due to fragmented ecosystems and labor-intensive, non-reproducible deployment workflows. Method: This paper introduces the first modular, reproducible, and extensible open-source automated benchmarking framework for eAI. It unifies model generation, optimization, conversion, and edge deployment, and integrates a standardized cross-tool, cross-model, and cross-hardware (six MCU families) benchmark suite evaluating latency, memory footprint, and accuracy. Contribution/Results: The framework enables the first systematic performance–accuracy–efficiency trade-off analysis of four major eAI toolchains—TFLite Micro, Edge Impulse, Arm Keil MDK, and TensorFlow Lite for Microcontrollers—across twelve representative models. Evaluation time is reduced from weeks to hours. All source code, CI/CD pipelines, and benchmark datasets are publicly released and widely adopted by the TinyML community.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Humans and AI: Human-in-the-loop Machine Learning
• Philosophy and Ethics of AI: Safety, Robustness & Trustworthiness

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications
• Economics, Online Markets and Human Computation: Economic ramifications for generative AI infrastructure and applications
• Graph Algorithms and Modeling for the Web: Foundation models and LLMs for Web-related graphs

Abstract

The integration of artificial intelligence (AI) into embedded devices, a paradigm known as embedded artificial intelligence (eAI) or tiny machine learning (TinyML), is transforming industries by enabling intelligent data processing at the edge. However, the many tools available in this domain leave researchers and developers wondering which one is best suited to their needs. This paper provides a review of existing eAI tools, highlighting their features, trade-offs, and limitations. Additionally, we introduce EdgeMark, an open-source automation system designed to streamline the workflow for deploying and benchmarking machine learning (ML) models on embedded platforms. EdgeMark simplifies model generation, optimization, conversion, and deployment while promoting modularity, reproducibility, and scalability. Experimental benchmarking results showcase the performance of widely used eAI tools, including TensorFlow Lite Micro (TFLM), Edge Impulse, Ekkono, and Renesas eAI Translator, across a wide range of models, revealing insights into their relative strengths and weaknesses. The findings provide guidance for researchers and developers in selecting the most suitable tools for specific application requirements, while EdgeMark lowers the barriers to adoption of eAI technologies.

Problem

• Benchmarking embedded AI tools
• Simplifying ML model deployment
• Comparing eAI tool performance

Innovation

• Automates ML model deployment
• Benchmarks eAI tool performance
• Promotes modularity and scalability