JaneEye: A 12-nm 2K-FPS 18.9-$mu$J/Frame Event-based Eye Tracking Accelerator

2025-09-18

AI Summary

To address the stringent requirements of high accuracy, low latency, and high energy efficiency for eye-tracking in XR wearable devices, this work proposes a dedicated event-camera-based accelerator. Methodologically, it replaces conventional frame-based inputs with sparse, high-temporal-resolution event streams; introduces a lightweight ConvJANET layer—retaining only the forget gate—to halve computational cost while preserving temporal modeling capability; and integrates hardware-aware optimizations including hardsigmoid/hardtanh linear approximations, fixed-point quantization, and ASIC co-design. Experimental results on the 3ET+ dataset demonstrate a localization error of 2.45 pixels using only 17.6K parameters, supporting event input rates up to 1250 Hz. The ASIC implementation achieves an end-to-end latency of 0.5 ms (equivalent to 2000 FPS) at 400 MHz and an energy efficiency of 18.9 μJ/frame—significantly outperforming state-of-the-art frame-based approaches.

Technology Category

• Computer Vision: Motion & Tracking
• Machine Learning: Hardware-aware ML
• Cognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

• Search and Retrieval-Augmented AI: Personalized, context-aware and across-device search
• User Modeling, Personalization and Recommendation: On-Device user modeling, personalization, and recommendation
• Responsible Web: Data and user privacy-enhancing technologies for the Web

Abstract

Eye tracking has become a key technology for gaze-based interactions in Extended Reality (XR). However, conventional frame-based eye-tracking systems often fall short of XR's stringent requirements for high accuracy, low latency, and energy efficiency. Event cameras present a compelling alternative, offering ultra-high temporal resolution and low power consumption. In this paper, we present JaneEye, an energy-efficient event-based eye-tracking hardware accelerator designed specifically for wearable devices, leveraging sparse, high-temporal-resolution event data. We introduce an ultra-lightweight neural network architecture featuring a novel ConvJANET layer, which simplifies the traditional ConvLSTM by retaining only the forget gate, thereby halving computational complexity without sacrificing temporal modeling capability. Our proposed model achieves high accuracy with a pixel error of 2.45 on the 3ET+ dataset, using only 17.6K parameters, with up to 1250 Hz event frame rate. To further enhance hardware efficiency, we employ custom linear approximations of activation functions (hardsigmoid and hardtanh) and fixed-point quantization. Through software-hardware co-design, our 12-nm ASIC implementation operates at 400 MHz, delivering an end-to-end latency of 0.5 ms (equivalent to 2000 Frames Per Second (FPS)) at an energy efficiency of 18.9 $mu$J/frame. JaneEye sets a new benchmark in low-power, high-performance eye-tracking solutions suitable for integration into next-generation XR wearables.

Problem

• Overcoming limitations of frame-based eye tracking for XR applications
• Achieving high accuracy and low latency with minimal energy consumption
• Enabling efficient eye tracking integration in wearable XR devices

Innovation

• Uses event cameras for high temporal resolution eye tracking
• Introduces lightweight ConvJANET with simplified forget gate architecture
• Implements custom activation approximations and fixed-point quantization