Aerial Gym Simulator: A Framework for Highly Parallelized Simulation of Aerial Robots

2025-03-03

AI Summary

This work addresses key challenges in multirotor UAV simulation—low computational efficiency, coarse-grained sensor annotation, and poor sim-to-real transferability—by introducing the first highly parallel, modular simulation and rendering framework for multirotor systems. Methodologically, it builds a parallel physics simulator on Isaac Gym supporting underactuated, fully actuated, and overactuated configurations; pioneers a GPU-accelerated geometric controller and a custom ray-tracing renderer enabling vertex-level environmental annotations (depth, semantic segmentation, surface normals); and deeply integrates CUDA and PyTorch to provide native reinforcement learning training interfaces. Contributions include achieving real-time simulation throughput exceeding 1,000 environments per second; enabling end-to-end sim-to-real policy transfer, validated in depth-aware navigation tasks on physical hardware; and open-sourcing the framework, which has since gained broad adoption within the robotics research community.

Technology Category

• Intelligent Robots: Multimodal Perception & Sensor Fusion
• Computer Vision: Multi-modal Vision
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• Systems and Infrastructure for Web, Mobile and WoT: Applied ML and AI for Web-based mobile applications

Abstract

This paper contributes the Aerial Gym Simulator, a highly parallelized, modular framework for simulation and rendering of arbitrary multirotor platforms based on NVIDIA Isaac Gym. Aerial Gym supports the simulation of under-, fully- and over-actuated multirotors offering parallelized geometric controllers, alongside a custom GPU-accelerated rendering framework for ray-casting capable of capturing depth, segmentation and vertex-level annotations from the environment. Multiple examples for key tasks, such as depth-based navigation through reinforcement learning are provided. The comprehensive set of tools developed within the framework makes it a powerful resource for research on learning for control, planning, and navigation using state information as well as exteroceptive sensor observations. Extensive simulation studies are conducted and successful sim2real transfer of trained policies is demonstrated. The Aerial Gym Simulator is open-sourced at: https://github.com/ntnu-arl/aerial_gym_simulator.

Problem

• Simulation of multirotor platforms with parallelized geometric controllers.
• GPU-accelerated rendering for depth, segmentation, and vertex-level annotations.
• Facilitates research on control, planning, and navigation using sensor data.

Innovation

• Highly parallelized simulation framework for multirotors
• GPU-accelerated rendering for depth and segmentation
• Supports reinforcement learning for depth-based navigation