Event-Based Upper-Body Humanoid Teleoperation Under Challenging Illumination

2026-07-31

AI Summary

This work addresses the failure of upper-body motion tracking with conventional RGB cameras under high dynamic range or extremely low-light conditions (<5 lux), where fixed exposure settings lead to severe performance degradation. To overcome this limitation, we present the first neuromorphic event-based teleoperation system for upper-body control. Our approach integrates an optimized event accumulation scheme, gravity-aligned inertial measurements, and a causal motion retargeting module named TWIST, all deployed on an NVIDIA Jetson platform to achieve end-to-end photon-to-action latency of only 23–34 milliseconds. Experimental results demonstrate that the proposed method substantially outperforms RGB-based baselines in challenging scenarios such as strong backlighting and near-dark environments, thereby validating the robustness and practicality of event cameras for fast and low-light teleoperation tasks.

Abstract

We present a real-time upper-body human-to-humanoid motion imitation framework driven by neuromorphic event-based vision. This work addresses practical perceptual bottlenecks of conventional frame-based RGB sensors, specifically their difficulty in high dynamic range (HDR) scenes and rapid motions due to fixed integration times. By leveraging the Prophesee EVK4 event camera, which operates asynchronously with high temporal resolution and a dynamic range exceeding 120 dB, our system supports stable tracking in conditions where standard vision pipelines degrade, such as severe backlighting and very low light environments below 5 lux. The architecture integrates a low-latency Perception Module, utilizing optimized event accumulation and gravity-aligned inertial fusion, with a causal Motion Module (TWIST) that performs online kinematic retargeting. We validate the system on an embedded NVIDIA Booster T1 platform and an 18-DoF humanoid upper-body setup, demonstrating an end-to-end photon-to-action latency of 23-34 ms and advantages over RGB baselines under our experimental setup. The results indicate a practical trade-off: events can be preferable for fast or poorly lit upper-body teleoperation, whereas well-lit static scenes may favor RGB or hybrid sensing.

Problem

• event-based vision
• humanoid teleoperation
• high dynamic range
• low-light conditions
• motion tracking

Innovation

• event-based vision
• humanoid teleoperation
• high dynamic range
• low-latency perception
• kinematic retargeting