StableTracker: Learning to Stably Track Target via Differentiable Simulation

2025-09-17

AI Summary

FPV object tracking suffers from hardware overload and error accumulation due to reliance on hand-crafted modular components—especially under high-speed, variable-velocity target motion. To address this, we propose an end-to-end differentiable control framework that jointly optimizes visual center alignment and fixed relative distance via differentiable physics simulation and backpropagation-through-time. The method integrates first-person-view (FPV) visual input, dynamic trajectory prediction, and deep reinforcement learning—eliminating the need for explicit state estimation or motion planning modules. In simulation, our approach consistently outperforms conventional geometric controllers and state-of-the-art learning-based baselines across diverse safety distances, trajectory shapes, and target velocities in terms of tracking accuracy and stability. Real-world experiments on a quadrotor platform further validate its feasibility, robustness, and generalization capability under unseen conditions.

Technology Category

• Computer Vision: Motion & Tracking
• Intelligent Robots: State Estimation
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: Federated recommendation systems and personalization
• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

FPV object tracking methods heavily rely on handcraft modular designs, resulting in hardware overload and cumulative error, which seriously degrades the tracking performance, especially for rapidly accelerating or decelerating targets. To address these challenges, we present extbf{StableTracker}, a learning-based control policy that enables quadrotors to robustly follow the moving target from arbitrary perspectives. The policy is trained using backpropagation-through-time via differentiable simulation, allowing the quadrotor to maintain the target at the center of the visual field in both horizontal and vertical directions, while keeping a fixed relative distance, thereby functioning as an autonomous aerial camera. We compare StableTracker against both state-of-the-art traditional algorithms and learning baselines. Simulation experiments demonstrate that our policy achieves superior accuracy, stability and generalization across varying safe distances, trajectories, and target velocities. Furthermore, a real-world experiment on a quadrotor with an onboard computer validated practicality of the proposed approach.

Problem

• Addresses unstable FPV object tracking for quadrotors
• Reduces hardware overload and cumulative tracking errors
• Enables robust target following from arbitrary perspectives

Innovation

• Learning-based control policy via differentiable simulation
• Backpropagation-through-time training for stable tracking
• Autonomous aerial camera with fixed relative distance