Knowledge Distillation for Reservoir-based Classifier: Human Activity Recognition

2025-05-29

AI Summary

To address the challenge of balancing accuracy and energy efficiency in real-time human activity recognition (HAR) at the edge, this paper proposes PatchEchoClassifier—a lightweight temporal classifier. Our method innovatively integrates knowledge distillation into the reservoir computing framework, employing an MLP-Mixer as the teacher model and a block-wise echo state network (ESN) as the student. Leveraging a novel patch-based tokenization scheme tailored for 1D sensor signals, it enables efficient temporal feature modeling. To the best of our knowledge, this is the first work to achieve deep structural integration of ESNs with knowledge transfer. Evaluated on multiple HAR benchmarks, PatchEchoClassifier achieves over 80% accuracy while consuming only 1/6 the FLOPs of DeepConvLSTM, significantly improving energy efficiency and inference latency. This work establishes a new paradigm for low-power edge-based time-series classification.

Technology Category

• Machine Learning: Learning on the Edge & Model Compression
• Humans and AI: Human-in-the-loop Machine Learning
• Planning, Routing, and Scheduling: Activity and Plan Recognition

Application Category

• Graph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs
• Economics, Online Markets and Human Computation: Data quality aspects of human-annotated datasets
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation

Abstract

This paper aims to develop an energy-efficient classifier for time-series data by introducing PatchEchoClassifier, a novel model that leverages a reservoir-based mechanism known as the Echo State Network (ESN). The model is designed for human activity recognition (HAR) using one-dimensional sensor signals and incorporates a tokenizer to extract patch-level representations. To train the model efficiently, we propose a knowledge distillation framework that transfers knowledge from a high-capacity MLP-Mixer teacher to the lightweight reservoir-based student model. Experimental evaluations on multiple HAR datasets demonstrate that our model achieves over 80 percent accuracy while significantly reducing computational cost. Notably, PatchEchoClassifier requires only about one-sixth of the floating point operations (FLOPS) compared to DeepConvLSTM, a widely used convolutional baseline. These results suggest that PatchEchoClassifier is a promising solution for real-time and energy-efficient human activity recognition in edge computing environments.

Problem

• Develop energy-efficient classifier for time-series HAR
• Transfer knowledge from MLP-Mixer to lightweight reservoir model
• Reduce computational cost while maintaining over 80% accuracy

Innovation

• PatchEchoClassifier uses Echo State Network
• Knowledge distillation from MLP-Mixer teacher
• Tokenizes sensor signals for patch-level features