embedded deployment

Deploying complex AI models efficiently on resource-constrained edge devices (e.g., smartphones, IoT endpoints) remains challenging due to stringent latency, memory, and energy constraints. Method: This paper proposes a three-tier co-optimization paradigm—spanning data, model, and system layers—unifying cross-layer coordination mechanisms for the first time and bridging a critical methodological gap in end-to-cloud full-stack optimization. It integrates lightweight edge data preprocessing, model compression techniques (pruning, quantization, knowledge distillation), efficient inference runtimes (TensorFlow Lite, ONNX Runtime), and heterogeneous hardware acceleration (NPU/GPU/FPGA). Contribution/Results: We construct a reusable edge AI optimization roadmap, achieving an average 3.2× inference speedup and 78% memory reduction while preserving ≥95% of the original model accuracy.

To address the performance bottlenecks of machine learning (ML) accelerators under growing model sizes and stringent energy-efficiency constraints, this paper proposes the first enhanced Roofline model deeply co-designed for ML accelerator characteristics. Our method introduces *execution paradigm boundary analysis* and *energy-constrained performance upper-bound modeling*, unifying the quantification of computational intensity, memory hierarchy, and data layout effects on both performance and energy efficiency. By integrating ML workload feature extraction, architecture-level quantitative analysis, and a hardware-algorithm co-evaluation framework, we systematically identify performance bottlenecks across mainstream accelerators for diverse operators and memory layouts. Experimental results reveal synergistic optimization pathways between memory bandwidth and computational density, and clarify several open research directions. The proposed model provides both theoretical foundations and practical guidance for energy-aware ML accelerator architecture design.

This work addresses the lack of systematic methodologies in model optimization, which often relies on heuristic choices and struggles to accommodate diverse deployment constraints. It formalizes model compression and acceleration as a constraint-aware multi-objective engineering decision problem, establishing a unified and actionable framework grounded in five key dimensions: data availability, latency, memory footprint, accuracy tolerance, and retraining budget. By integrating techniques such as quantization, pruning, knowledge distillation, parameter-efficient fine-tuning (PEFT), and inference optimization, the study proposes tailored optimization pipelines for four representative industrial scenarios, delivering a reproducible and quantifiable guide for technology selection.

This study addresses the stringent constraints on size, power, and computational resources faced by AI inference on resource-limited platforms such as small satellites. By conducting empirical characterization of quantized AI inference on Cortex-M-class processors using representative embedded vision neural networks, the work establishes the first measurement-based performance baseline for on-board embedded systems. It introduces an explicit multi-core/multi-device cooperative scheduling mechanism and integrates analysis of ALU/SIMD utilization with memory traffic to evaluate system behavior. Moving beyond conventional paradigms that rely on opaque OS-level scheduling, this research provides comparable latency and data-movement benchmarks for typical spaceborne processors like LEON and NOEL-V, thereby demonstrating the critical role of architecture-aware design and cooperative scheduling as key dimensions in optimizing embedded AI inference for satellite applications.

Deploying the Qwen2.5-0.5B model on the Xilinx Kria KV260 edge platform (ARM Cortex-A53 + FPGA) faces critical bottlenecks in computational demand, memory bandwidth, and power consumption. To address these challenges, this work proposes a co-optimization framework integrating Activation-aware Weight Quantization (AWQ) with a custom FPGA pipeline, alongside a CPU–FPGA hybrid execution paradigm and dynamic offloading scheduling strategy. The approach jointly optimizes quantization and hardware acceleration to preserve model accuracy while significantly reducing resource requirements. Experimental results demonstrate a 55.08% model compression rate and an inference throughput of 5.1 tokens/s—82% higher than the baseline. This yields substantial improvements in deployment efficiency and energy efficiency for large language models on resource-constrained edge devices.

Modern large-scale distributed training faces sharply diminishing returns in hardware scaling: as GPU counts reach thousands, communication overhead dominates performance bottlenecks, rendering conventional parallelism strategies—data, tensor, and pipeline parallelism—suboptimal. Method: Leveraging real-world LLM training workloads, this project establishes an empirical analytical framework spanning diverse model scales, hardware configurations, and parallelization strategies. It quantifies the nonlinear relationship between accelerator count and performance gain, precisely identifying critical inflection points across model, data, and compute scaling dimensions. Contribution/Results: We discover that low-communication “suboptimal” strategies become optimal at extreme scale; we empirically determine hardware selection criteria, cluster topology requirements, and optimal parallelism combinations for training billion-parameter models. Our findings provide actionable, deployment-ready optimization guidelines for trillion-parameter LLM training infrastructures.

Existing scientific computing codes are difficult to efficiently port to specialized architectures such as AMD AI Engine, often requiring extensive manual refactoring. This work proposes a tensor abstraction–based compilation approach that automatically elevates generic loops to tensor semantics by parsing lightweight OpenMP annotations, and constructs an end-to-end compilation pipeline to map computations onto the AI Engine execution model. The method significantly reduces programming complexity through minimal OpenMP directives and enables CPU–NPU cooperative scheduling. Experimental results on six scientific and AI kernel benchmarks show that the NPU achieves higher energy efficiency than a multi-core CPU at float32 precision; for two kernels, cooperative execution yields a 40% performance improvement and 15% energy reduction.

This work addresses the challenges of SLO violations and resource inefficiency in machine learning model serving caused by inadequate capacity planning. To this end, the authors propose an adaptive, feedback-driven load testing framework that formalizes the ML serving load testing process for the first time. The framework incorporates real-traffic-based workload calibration and a warm-up mechanism, combined with adaptive search, performance signal feedback control, convergence detection, and GPU monitoring to efficiently estimate the maximum sustainable throughput under SLO constraints. Evaluation across 14 industrial cases demonstrates that the approach reduces capacity estimation error from approximately 30% to 2–6%, with the warm-up mechanism improving accuracy by 22.2%. This significantly mitigates deployment incidents and enhances GPU resource utilization efficiency.

This work addresses the challenge of deploying low-latency, small-batch neural networks with fully on-chip weight storage in extreme-edge scientific computing, where traditional programmable logic falls short. The study systematically evaluates the inference performance gap between AI Engines and programmable logic, proposing spatiotemporal dataflow optimizations tailored for AI Engines. It introduces a novel metric—Latency-Adjusted Resource Equivalence (LARE)—to rigorously delineate, for the first time, the operational regimes where AI Engines outperform programmable logic. Through comprehensive architectural analysis, microbenchmarking, spatial and API-level optimizations, and integration into the hls4ml toolchain, the authors successfully deploy multiple end-to-end neural networks, demonstrating the scalability and performance advantages of AI Engines under stringent resource constraints.

This study addresses the critical challenge of ensuring accuracy, stability, and clinical real-time performance in edge AI medical devices, where existing validation methods inadequately cover temporal safety at the inference layer. On an NVIDIA Jetson Orin Nano Super platform, the authors empirically compare the temporal behavior of a MobileNetV2 model executed via TensorRT FP16 on GPU versus ONNX Runtime FP32 on CPU. Despite both configurations achieving zero Software-Triggered Error Rate (STER), the GPU pipeline maintains end-to-end latency consistently below 11 ms—meeting the 10 Hz clinical cycle requirement—whereas the CPU path exhibits 9.8× higher average latency and exceeds timing bounds by 65%. These findings reveal a structural independence among accuracy, stability, and temporal constraints, prompting the proposal of a novel validation paradigm that jointly assesses STER and latency to fulfill FDA regulatory expectations for inference-layer robustness.