Static In, Dynamic Out: Counterfactual Action Augmentation for Moving Object Manipulation

2026-07-30

AI Summary

This work addresses the limitation of existing visuomotor policies, which typically assume static target objects and consequently suffer significant performance degradation when manipulating dynamic objects—such as parts on a conveyor belt or wind-swaying fruits. To overcome this, the authors propose SIDO, a method that decouples dynamic object manipulation into two subtasks: future pose prediction and goal-conditioned action generation. SIDO further introduces a counterfactual action augmentation mechanism, enabling policies trained solely on static demonstrations to achieve zero-shot generalization to unseen object motion patterns. By leveraging a target pose predictor and action chunk deformation, the approach preserves hand-object relative pose consistency. Evaluated across three simulated tasks and two real-world scenarios involving five distinct dynamic modes, SIDO substantially improves manipulation success rates while maintaining original performance on static objects.

Abstract

Visuomotor policies have advanced on manipulation tasks where the target object stays static during execution, but real deployments break this assumption: parts drift on conveyors and fruits sway in the wind. We introduce Static In, Dynamic Out (SIDO), a counterfactual action augmentation that enables a policy trained only on static object demonstrations to adapt to unseen object motion at test time. Our key idea is to factorize moving object manipulation into two sub-problems: predicting where the object will be, and reaching that predicted pose. SIDO displaces the object to a counterfactual future position and morphs the demonstrated action chunk to preserve the hand-object relative pose, yielding a goal-conditioned policy. At deployment an object pose predictor supplies the future position. Across three simulated tasks (Mug, Square, Stack) under five object motion patterns and two real-world tasks (Gantry, Peachtree), SIDO improves moving object success over the baselines while preserving static object performance. Project website: https://sido-staticindynamicout.github.io/.

Problem

• moving object manipulation
• visuomotor policies
• object motion
• static assumption
• real-world deployment

Innovation

• counterfactual action augmentation
• visuomotor policy
• moving object manipulation
• goal-conditioned policy
• object pose prediction