DiLQR: Differentiable Iterative Linear Quadratic Regulator via Implicit Differentiation

2025-06-20

AI Summary

This work addresses the scalability limitation of iterative Linear-Quadratic Regulator (iLQR) in differentiable optimal control, where conventional automatic differentiation (AD) suffers from prohibitive memory and computational costs due to explicit unrolling of iterative trajectories. We propose an implicit-differentiation-based method to compute analytical gradients of iLQR, achieving the first efficient, exact, and iteration-number-invariant differentiable iLQR. By bypassing trajectory unrolling, our approach drastically reduces memory footprint and backward-pass complexity. Integrated as an end-to-end trainable module, the differentiable iLQR seamlessly embeds into neural architectures—e.g., vision-control joint models—enabling gradient propagation through the entire control pipeline. Experiments on benchmark control tasks demonstrate up to 128× faster backpropagation versus AD, six orders-of-magnitude improvement in policy learning efficiency, and significantly lower convergence loss compared to AD-based baselines. This work establishes a theoretically rigorous and practically deployable paradigm for differentiable optimal control.

Technology Category

• Search and Optimization: Learning to Search
• Intelligent Robots: Learning & Optimization for ROB
• Machine Learning: Imitation Learning & Inverse Reinforcement Learning

Application Category

• Search and Retrieval-Augmented AI: Web learning to rank, online learning, and counterfactual learning for ranking
• Economics, Online Markets and Human Computation: LLM based quality controls for crowd work
• Responsible Web: Machine-in-the-loop, human agency and autonomy

Abstract

While differentiable control has emerged as a powerful paradigm combining model-free flexibility with model-based efficiency, the iterative Linear Quadratic Regulator (iLQR) remains underexplored as a differentiable component. The scalability of differentiating through extended iterations and horizons poses significant challenges, hindering iLQR from being an effective differentiable controller. This paper introduces DiLQR, a framework that facilitates differentiation through iLQR, allowing it to serve as a trainable and differentiable module, either as or within a neural network. A novel aspect of this framework is the analytical solution that it provides for the gradient of an iLQR controller through implicit differentiation, which ensures a constant backward cost regardless of iteration, while producing an accurate gradient. We evaluate our framework on imitation tasks on famous control benchmarks. Our analytical method demonstrates superior computational performance, achieving up to 128x speedup and a minimum of 21x speedup compared to automatic differentiation. Our method also demonstrates superior learning performance ($10^6$x) compared to traditional neural network policies and better model loss with differentiable controllers that lack exact analytical gradients. Furthermore, we integrate our module into a larger network with visual inputs to demonstrate the capacity of our method for high-dimensional, fully end-to-end tasks. Codes can be found on the project homepage https://sites.google.com/view/dilqr/.

Problem

• Enabling iLQR as a differentiable controller via implicit differentiation
• Solving scalability challenges in differentiating iLQR iterations and horizons
• Integrating DiLQR into neural networks for end-to-end learning tasks

Innovation

• DiLQR enables differentiable iLQR via implicit differentiation
• Analytical gradient solution ensures constant backward cost
• Achieves up to 128x speedup over automatic differentiation