FlowPilot: Real-Time World-Action Modeling for Agile UAV Navigation

2026-08-01

AI Summary

This work addresses the challenge of high-speed, real-time onboard navigation for drones in complex environments, where reliance on map reconstruction or lack of explicit scene prediction often hinders performance. To this end, we propose FlowPilot—a compact joint world-action model that, for the first time, leverages flow matching to simultaneously denoise future depth observations and executable trajectories. Our approach employs a dual-stream hybrid Transformer that couples visual and action experts, and represents trajectories using C²-continuous Bernstein polynomials to ensure smoothness and trackability. Trained via a three-stage depth pyramid—from high-throughput simulation to photorealistic simulation and finally real onboard data—FlowPilot outperforms existing learning- and optimization-based baselines in closed-loop simulation, enabling agile flight at speeds up to 8 m/s. On a Jetson Orin NX, it achieves under 18 ms latency and demonstrates real-world autonomous navigation at 5.5 m/s using only onboard sensing.

Abstract

We present FlowPilot, a compact world-action model for real-time onboard UAV navigation from depth. Unlike map-then-optimize pipelines that require local reconstruction or end-to-end policies that lack explicit scene prediction, FlowPilot jointly denoises future depth observations and executable trajectories with flow matching. A dual-stream mixture-of-transformers couples video and action experts through shared attention, allowing future-scene prediction and trajectory generation to inform each other. At deployment, the model runs action-centrically and outputs only a trajectory. To ensure trackability, actions are parameterized as degree-7 Bernstein polynomials: the current state constrains the initial control points, and the network predicts five free control points, yielding C^2-continuous references with closed-form velocity, acceleration and jerk. FlowPilot is trained on a three-level depth pyramid spanning high-throughput simulation, photorealistic simulation, and real onboard data. In closed-loop simulation, it outperforms learning- and optimization-based baselines under increasing clutter and commanded speeds up to 8m/s. On a physical quadrotor, the full perception-to-action pipeline runs in under 18ms on a Jetson Orin NX and reaches 5.5m/s in cluttered indoor and forest environments using only onboard sensing and computation.

Problem

• UAV navigation
• real-time onboard perception
• agile flight
• depth-based navigation
• trajectory tracking

Innovation

• flow matching
• dual-stream mixture-of-transformers
• Bernstein polynomial trajectory
• onboard UAV navigation
• real-time world-action modeling