Patch Policy: Efficient Embodied Control via Dense Visual Representations

2026-07-20

AI Summary

This work addresses the limitations of existing robotic policies that either lose fine-grained spatial details through global pooling or incur prohibitive computational costs when leveraging pretrained visual features. The authors propose a lightweight Transformer-based policy architecture that, for the first time, enables efficient utilization of dense patch-level features from Vision Transformers (ViTs) without requiring finetuning of large models. By incorporating blockwise causal attention, the method preserves temporal causality while effectively fusing multi-patch visual inputs with agent state information. Despite using only approximately 0.7% of the parameters of OpenVLA-OFT, the proposed approach achieves a 40% performance gain over global pooling baselines and outperforms the finetuned OpenVLA-OFT by 18% across four simulated and three real-world robotic environments.

Abstract

Pretrained dense visual features from Vision Transformers (ViTs) are powerful yet have been underutilized in robot learning. Modern robot policies either compress each observation into a single global token, or rely on visual backbones trained from scratch, sacrificing both fine-grained spatial detail and the benefits of large-scale visual pre-training. While there exist policies that do operate on dense patch features like large vision-language-action models (VLAs), they tend to be heavy and slow, inheriting the full cost of a billion-parameter vision-language model (VLM) backbone. We close this gap with Patch Policy, a minimal architectural extension that enables transformer-based policies to consume dense pre-trained patch tokens directly without the computational overhead of a full VLM. At its core is a block-causal attention mask that preserves the temporal causality of standard policies while letting the model attend over many patch tokens per observation, alongside other state information. Patch Policy is lightweight, fast, and highly effective. Across four simulated and three real-world environment suites, our method achieves a 40% relative improvement over policies using state-of-the-art global-pooled representations. Furthermore, it surpasses fine-tuned OpenVLA-OFT by 18% while using roughly 0.7% of the parameters. We believe Patch Policy provides a pipeline for the robotics community to readily leverage continuing progress in visual representation learning, without sacrificing the training efficiency or inference speed required for high-frequency, reactive control. Videos can be viewed at https://patch-policy.github.io

Problem

• embodied control
• dense visual representations
• robot learning
• Vision Transformers
• visual pre-training

Innovation

• Patch Policy
• dense visual representations
• Vision Transformers
• block-causal attention
• embodied control