Privacy-preserving fall detection at the edge using Sony IMX636 event-based vision sensor and Intel Loihi 2 neuromorphic processor

2025-11-27

AI Summary

To address privacy sensitivity, edge deployment constraints, and energy efficiency limitations in home-based fall detection for elderly individuals, this work proposes an event-driven, neuromorphic computing–enabled real-time on-device detection framework. Methodologically, it integrates the Sony IMX636 event camera with the Intel Loihi 2 neuromorphic processor, implementing a hierarchical spiking convolutional SNN based on leaky integrate-and-fire (LIF) neurons. It innovatively incorporates the MCUNet lightweight backbone and the S4D state-space module to enable efficient spatiotemporal modeling of asynchronous event streams. Experiments demonstrate that the system achieves an 84% F1 score on Loihi 2 while consuming only 90 mW—reducing spike operations by 5× and increasing sparsity to 2×, outperforming baselines by +6% F1. This work is the first to empirically validate the synergistic effectiveness of event-driven sensing, state-space modeling, and spiking neural networks under extreme resource constraints at the edge, establishing a scalable paradigm for privacy-preserving, ultra-low-power geriatric health monitoring.

Technology Category

• Cognitive Modeling & Cognitive Systems: Neural Spike Coding
• Machine Learning: Learning on the Edge & Model Compression
• Intelligent Robots: Multimodal Perception & Sensor Fusion

Application Category

• Systems and Infrastructure for Web, Mobile and WoT: Energy management for devices in mobile Web and WoT environments
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Security and Privacy: Large-scale security measurements

Abstract

Fall detection for elderly care using non-invasive vision-based systems remains an important yet unsolved problem. Driven by strict privacy requirements, inference must run at the edge of the vision sensor, demanding robust, real-time, and always-on perception under tight hardware constraints. To address these challenges, we propose a neuromorphic fall detection system that integrates the Sony IMX636 event-based vision sensor with the Intel Loihi 2 neuromorphic processor via a dedicated FPGA-based interface, leveraging the sparsity of event data together with near-memory asynchronous processing. Using a newly recorded dataset under diverse environmental conditions, we explore the design space of sparse neural networks deployable on a single Loihi 2 chip and analyze the tradeoffs between detection F1 score and computational cost. Notably, on the Pareto front, our LIF-based convolutional SNN with graded spikes achieves the highest computational efficiency, reaching a 55x synaptic operations sparsity for an F1 score of 58%. The LIF with graded spikes shows a gain of 6% in F1 score with 5x less operations compared to binary spikes. Furthermore, our MCUNet feature extractor with patched inference, combined with the S4D state space model, achieves the highest F1 score of 84% with a synaptic operations sparsity of 2x and a total power consumption of 90 mW on Loihi 2. Overall, our smart security camera proof-of-concept highlights the potential of integrating neuromorphic sensing and processing for edge AI applications where latency, energy consumption, and privacy are critical.

Problem

• Develops a privacy-preserving fall detection system for elderly care using neuromorphic sensors and processors.
• Addresses real-time edge inference under strict hardware constraints with low power consumption.
• Explores trade-offs between detection accuracy and computational efficiency in sparse neural networks.

Innovation

• Event-based vision sensor for privacy-preserving fall detection
• Neuromorphic processor with sparse neural networks for efficiency
• FPGA interface enabling near-memory asynchronous processing