FM-VLA: Force-based Memory for Vision-Language-Action Models in Contact-Rich Manipulation

2026-07-20

AI Summary

This work addresses the challenge faced by vision-language-action (VLA) models in non-Markovian, contact-intensive tasks, where visual ambiguity hinders effective use of historical context. To overcome this limitation, the study introduces force history as a novel memory modality into the VLA framework—a first in the field. A pretrained variational autoencoder (VAE) compresses temporal force signals into compact memory tokens, which are fused with short-term state histories to condition the action policy module. This integration significantly enhances the model’s temporal reasoning about contact events. Evaluated on three memory-dependent manipulation tasks, the approach achieves over 80% success rates, substantially outperforming existing baselines while incurring minimal inference overhead.

Abstract

Vision-language-action (VLA) models have achieved impressive generalization in robotic manipulation, and recent memory-augmented VLAs have relaxed the Markovian assumption by conditioning on past images or language summaries. Vision-based memory approaches address this by conditioning on sampled past image frames, but they are computationally expensive and fundamentally limited when temporal events are visually ambiguous, e.g., pushing a button multiple times with small movements. We propose FM-VLA, a VLA model with force-based memory, enabling temporal context reasoning for non-Markovian, contact-rich manipulation. We encode force histories into compact force memory tokens with a variational autoencoder (VAE) pretrained with force time series reconstruction. By projecting force latent representations and short state history as additional conditioning tokens to the action expert module, we enable VLAs to leverage accumulated contact event history to guide manipulation. We evaluate FM-VLA on three memory-dependent tasks, including finding a hidden block, pressing a button, and wiping a dish for a specific number of times. Our lightweight force memory achieves over 80% success rate with minimal inference overhead, significantly outperforming baseline approaches. Project page: https://qft-333.github.io/FM-VLA-Page/

Problem

• Vision-Language-Action
• contact-rich manipulation
• non-Markovian
• force-based memory
• temporal context

Innovation

• force-based memory
• vision-language-action models
• non-Markovian manipulation
• variational autoencoder
• contact-rich tasks