MAUPITI: On-Device Prototype-Based Learning on a Smart Infrared Sensor

2026-08-07

AI Summary

This work addresses privacy-preserving online continual learning of human pose and gestures on ultra-low-power embedded infrared sensors with less than 32 kB of memory. It proposes a replay-free streaming prototype update mechanism that combines an offline-trained, quantized CNN encoder with a nearest-class-mean (NCM) classifier to dynamically maintain class prototypes on-device, thereby eliminating the need for backpropagation and replay buffers. Implemented on a RISC-V microcontroller equipped with low-precision SIMD extensions and a 16×16 TMOS infrared array, the system achieves accuracy comparable to conventional methods on two datasets, with a total latency overhead below 0.29% and power consumption of approximately 1.5 mW—demonstrating, for the first time, efficient, low-latency online continual learning on intelligent infrared sensors.

Abstract

Low-resolution infrared (IR) array sensors represent an interesting solution for privacy-preserving human sensing in embedded systems. In this letter, we describe a smart multi-pixel IR sensor integrating a 16$\times$16 thermal MOSFET (TMOS) array and a RISC-V microcontroller extended with low-precision SIMD instructions, capable of on-device learning and continual adaptation for pose and gesture recognition tasks under tight memory and power constraints ($<$32kB on-chip memory, $\approx$1.5mW). To avoid the memory overheads of backpropagation and replay buffers, we adopt a prototype-based Nearest Class Mean (NCM) classifier in which a simple Convolutional Neural Network (CNN) encoder is trained and quantized offline, while class prototypes are stored and updated on the device in streaming mode. With experiments on two datasets, we show that this approach yields accuracy on par with a conventional classifier, with negligible latency overheads in both the classification and the prototype update ($<$0.29% considering both phases), effectively enabling online adaptation of the perception framework.

Problem

• on-device learning
• infrared sensor
• prototype-based classification
• continual adaptation
• resource-constrained embedded systems

Innovation

• on-device learning
• prototype-based classification
• low-power sensing
• RISC-V SIMD
• thermal infrared array