GoStop: Reinforcement Learning for Adaptive Temporal Aggregation in Event-Based Feature Tracking

2026-07-17

AI Summary

This work addresses the limitations of existing event-based feature tracking methods, which rely on fixed heuristic rules for event accumulation and struggle to adapt to complex motion scenarios involving either rapid or slow dynamics. To overcome this, we propose a novel reinforcement learning–based adaptive temporal aggregation framework that formulates event accumulation as a sequential decision-making problem, dynamically determining whether to accumulate events or trigger tracking inference based on real-time motion cues. Our approach eliminates the need for handcrafted rules and can be seamlessly integrated as a plug-in module into existing trackers. To facilitate robust evaluation under diverse dynamic conditions, we introduce DEFT, a new dataset encompassing a wide range of motion patterns. Experimental results demonstrate that our method significantly outperforms conventional heuristic strategies, achieving a superior balance between tracking accuracy and efficiency in dynamic scenes.

Abstract

Feature tracking plays a fundamental role in understanding scene motion and supports various downstream tasks. Event cameras, with their high temporal resolution and asynchronous sensing, enable low-latency and motion-robust perception, making them well-suited for feature tracking under fast and non-linear motion. However, existing event-based feature tracking methods rely on fixed heuristic rules based on hand-tuning for event accumulation. Such strategies fail to adapt to diverse motion dynamics, leading to degraded performance under abrupt motion changes or low-motion scenarios. In this paper, we model event accumulation as a sequential decision-making problem and introduce reinforcement learning (RL) framework to adaptively control the accumulation process for online event-based feature tracking. Our approach trains a RL agent that decides whether to continue accumulating events or to perform tracking inference based on motion cues. The proposed adaptive temporal agent enables dynamic adaptation to varying motion patterns without relying on hand-crafted rules. Furthermore, we introduce a Dynamic Event-based Tracking (DEFT) dataset with dynamic motion distributions to evaluate the robustness of the feature tracking. Extensive experiments demonstrate that integrating our plug-and-play framework to existing feature tracking methods consistently outperforms heuristic-based approaches, improving robustness under dynamic motion while offering a better balance between tracking accuracy and efficiency. Our project codes and datasets are available at https://github.com/kmax2001/GoSTOP

Problem

• event-based feature tracking
• temporal aggregation
• motion dynamics
• adaptive control
• heuristic rules

Innovation

• Reinforcement Learning
• Event-based Vision
• Adaptive Temporal Aggregation
• Feature Tracking
• Dynamic Motion