Estimate, Don't Imitate: Reusing Differentiable State-Based Policies for Visuomotor Control

2026-09-27

AI Summary

This study addresses the limitations of visual policies that directly imitate experts, which often suffer from redundant learning and fail to effectively correlate state estimation errors with control performance. To overcome these issues, this work proposes a framework that reuses a frozen differentiable state expert, training only a visual interface to reconstruct missing states for manipulation. An action consistency loss is introduced to directionally optimize state estimation errors that significantly impact downstream control via backpropagation. Furthermore, Sim-to-Real transfer is achieved by combining teacher-student distillation with hybrid supervision objectives. Across five tasks, the proposed method consistently outperforms pixel-level imitation learning baselines, achieving a 76% real-world success rate on a Panda robot without retraining.

Abstract

Simulation-trained manipulation policies can exploit privileged state information to learn effective contact-rich behaviours, but deployment requires acting from partial observations such as noisy camera images. A common solution is teacher-student distillation, in which a visuomotor policy is trained to reproduce the actions of the privileged expert. This requires the student to jointly infer the task-relevant state and relearn the expert's action mapping that is already available. An alternative is to reuse the state-based expert and learn only a perceptual interface that reconstructs its missing state inputs. However, minimising the state estimate error alone does not necessarily minimise the downstream control error induced by these estimates. To bridge this gap, we train a visual state estimator using both direct state supervision and an action-consistency loss backpropagated through the frozen, differentiable expert. A scheduled objective first establishes a physically meaningful state estimate and progressively emphasises errors that affect the expert's actions. Across five goal-conditioned manipulation tasks, retaining the expert consistently outperforms direct pixel-to-action imitation from the same expert demonstration corpus. We further demonstrate sim-to-real transfer on a physical Panda robot, achieving 76% success without retraining the underlying expert.

Problem

• visuomotor control
• state estimation
• teacher-student distillation
• privileged state information
• sim-to-real transfer

Innovation

• Visuomotor Control
• Differentiable Policy
• State Estimation
• Action-Consistency Loss
• Sim-to-Real Transfer