Empowering On-Device Model Adaptation with an Edge AI Inference Accelerator

2026-07-20

AI Summary

This work addresses the challenge of on-device model adaptation under resource constraints, where end-to-end backpropagation through modern deep networks is infeasible on edge hardware. The authors propose a heterogeneous adaptation pipeline that deploys an INT8-quantized backbone on a Hailo-8L inference accelerator for frozen feature extraction, while fine-tuning only a lightweight classification head in FP32 on the CPU. A post-training quantization recovery mechanism is introduced to preserve feature quality. This approach uniquely repurposes commercial inference accelerators for on-device training and overcomes training efficiency bottlenecks through co-design of the backbone and head. Experiments demonstrate up to a 15.4× speedup in training over a Raspberry Pi 5 CPU baseline, substantially reduced per-sample energy consumption, and competitive accuracy across multiple architectures and datasets.

Abstract

On-device model adaptation is essential to enable lifelong personalization on resource-constrained hardware, but compute, power, and memory limitations of such devices make end-to-end backpropagation impractical for modern deep neural networks. This work proposes a heterogeneous adaptation pipeline that repurposes a commercial edge AI inference accelerator, Hailo-8L, for frozen-backbone feature extraction during on-device training. The computational graph is partitioned so that the pre-trained backbone is quantized to INT8 and run on the accelerator, while only a lightweight FP32 classification head is fine-tuned on the host CPU, enabling frequent, energy-efficient in-field updates with most weights remaining fixed. Across multiple architectures and datasets, this pipeline achieves up to 15.4x faster wall-clock training time compared to a Raspberry Pi 5 CPU baseline, offers competitive throughput in favorable settings, and consistently reduces energy per sample. Post-training quantization restoration is shown to be crucial for preserving the quality of accelerator-generated features and mitigating accuracy loss in quantization-sensitive architectures. Overall, the results demonstrate a practical approach to efficient on-device adaptation using inference-oriented edge accelerators. The implementation is available at https://github.com/MatPiech/accelerator-training.

Problem

• on-device model adaptation
• edge AI
• resource-constrained hardware
• backpropagation
• model personalization

Innovation

• on-device adaptation
• edge AI accelerator
• heterogeneous training pipeline
• post-training quantization restoration
• frozen-backbone fine-tuning