Balance Equation-based Distributionally Robust Offline Imitation Learning

2025-11-11

AI Summary

This work addresses the robustness of imitation learning (IL) under dynamic shift, proposing an offline distributionally robust IL framework that requires no online interaction. Conventional IL methods assume consistent dynamics between training and deployment; however, modeling errors and environmental disturbances often cause severe performance degradation in practice. Our approach innovatively integrates distributionally robust optimization (DRO) with Bellman-type consistency constraints, optimizing worst-case performance over a predefined uncertainty set of transition dynamics—using only expert demonstrations collected under nominal dynamics. By avoiding additional environment sampling, the method achieves both theoretical interpretability and engineering practicality. Experiments across multiple continuous-control benchmarks demonstrate that our method significantly outperforms existing offline IL baselines, exhibiting superior robustness and generalization under dynamic perturbations.

Technology Category

• Machine Learning: Imitation Learning & Inverse Reinforcement Learning
• Intelligent Robots: Learning & Optimization for ROB
• Search and Optimization: Sampling/Simulation-based Search

Application Category

• Economics, Online Markets and Human Computation: Cost models of using LLMs in production systems
• Responsible Web: Machine-in-the-loop, human agency and autonomy
• User Modeling, Personalization and Recommendation: User modeling and simulation for interactive and conversational systems

Abstract

Imitation Learning (IL) has proven highly effective for robotic and control tasks where manually designing reward functions or explicit controllers is infeasible. However, standard IL methods implicitly assume that the environment dynamics remain fixed between training and deployment. In practice, this assumption rarely holds where modeling inaccuracies, real-world parameter variations, and adversarial perturbations can all induce shifts in transition dynamics, leading to severe performance degradation. We address this challenge through Balance Equation-based Distributionally Robust Offline Imitation Learning, a framework that learns robust policies solely from expert demonstrations collected under nominal dynamics, without requiring further environment interaction. We formulate the problem as a distributionally robust optimization over an uncertainty set of transition models, seeking a policy that minimizes the imitation loss under the worst-case transition distribution. Importantly, we show that this robust objective can be reformulated entirely in terms of the nominal data distribution, enabling tractable offline learning. Empirical evaluations on continuous-control benchmarks demonstrate that our approach achieves superior robustness and generalization compared to state-of-the-art offline IL baselines, particularly under perturbed or shifted environments.

Problem

• Addresses performance degradation from environmental dynamics shifts
• Learns robust policies solely from expert demonstrations offline
• Minimizes imitation loss under worst-case transition distributions

Innovation

• Distributionally robust optimization over transition models
• Reformulates robust objective using nominal data
• Learns robust policies from offline expert demonstrations