Going with the Flow: Koopman Behavioral Models as Implicit Planners for Visuo-Motor Dexterity

2026-04-11

AI Summary

Existing dexterous manipulation methods based on diffusion models or Transformers rely on fixed temporal action chunking, which struggles to balance temporal coherence with real-time responsiveness and incurs high computational costs. This work proposes a Unified Behavior Model (UBM) that, for the first time, applies Koopman operator theory to jointly model visual and proprioceptive flows, constructing linearized latent-space dynamics that inherently ensure temporal consistency and enable implicit planning without an explicit planner. Additionally, UBM incorporates an online replanning mechanism driven by visual flow prediction error. Evaluated across seven simulated and two real-world tasks, UBM matches or exceeds state-of-the-art performance while offering faster inference, smoother execution, robustness to occlusion, and flexible replanning capabilities.

Technology Category

Humans and AI: Human-Aware Planning and Behavior PredictionIntelligent Robots: ManipulationComputer Vision: Diffusion Models for Vision

Application Category

Semantics and Knowledge: Data modeling to support human-machine intelligence, including LLMs agents, intelligent system behavior, explanations, and user-friendly interactionsResponsible Web: Machine-in-the-loop, human agency and autonomyGraph Algorithms and Modeling for the Web: Efficient manipulation of static and dynamic Web-related graphs

Abstract

There has been rapid and dramatic progress in learning complex visuo-motor manipulation skills from demonstrations, thanks in part to expressive policy classes that employ diffusion- and transformer-based backbones. However, these design choices require significant data and computational resources and remain far from reliable, particularly within the context of multi-fingered dexterous manipulation. Fundamentally, they model skills as reactive mappings and rely on fixed-horizon action chunking to mitigate jitter, creating a rigid trade-off between temporal coherence and reactivity. In this work, we introduce Unified Behavioral Models (UBMs), a framework that learns to represent dexterous skills as coupled dynamical systems that capture how visual features of the environment (visual flow) and proprioceptive states of the robot (action flow) co-evolve. By capturing such behavioral dynamics, UBMs can ensure temporal coherence by construction rather than by heuristic averaging. To operationalize these models, we propose Koopman-UBM, a first instantiation of UBMs that leverages Koopman Operator theory to effectively learn a unified representation in which the joint flow of latent visual and proprioceptive features is governed by a structured linear system. We demonstrate that Koopman-UBM can be viewed as an implicit planner: given an initial condition, it computes the desired robot behavior with the resulting flow of visual features over the entire skill horizon. To enable reactivity, we introduce an online replanning strategy in which the model acts as its own runtime monitor that automatically triggers replanning when predicted and observed visual flow diverge. Across seven simulated and two real-world tasks, we demonstrate that K-UBM matches or exceeds the performance of SOTA baselines, while offering faster inference, smooth execution, robustness to occlusions, and flexible replanning.

Problem

• visuo-motor dexterity
• temporal coherence
• reactivity
• action chunking
• dexterous manipulation

Innovation

• Koopman Operator
• Behavioral Dynamics
• Implicit Planning
• Visual-Proprioceptive Flow
• Online Replanning