SmoLSTM: A Compact Vision-Language-Action Model with Recurrent Memory that Persists

2026-09-19

AI Summary

为解决视觉语言动作模型在处理遮挡或视觉相同物体时的模糊性问题，提出SmoLSTM模型，通过循环状态存储历史信息，并采用流匹配动作头预测动作。

Abstract

Vision-language-action models often predict actions from only the current observation, which can leave tasks involving object occlusion or visually identical objects ambiguous without episode history. The usual countermeasure, widening the observation window, turns the horizon into a hyperparameter and lets per-step cost grow with it. We instead capture the episode in a recurrent state. SmoLSTM couples a frozen 256M-parameter SmolVLM backbone to a matrix-memory LSTM control layer in which observation tokens and action queries are unified in a single causal stream that is never reset throughout the entire episode. Recurrent-state storage is therefore O(1) in episode length. A flow-matching action head predicts chunks of 10 end-effector pose deltas and gripper commands at each control step. Our single policy, trained jointly on 7,461 demonstrations across 140 tasks and evaluated on held-out initial states, performs best, reaching 85.1% subgoal coverage and 77.5% full-task success on LIBERO-Mem with 0.04B trainable parameters, surpassing both the benchmark's own object-centric baseline and recent memory-based approaches. Resetting the recurrent state at every control step reduces full-task success to 7.0%, showing that the trained policy relies on context carried between decisions. The same model achieves 79.6% average success on standard LIBERO.

Problem

• Vision-language-action models
• object occlusion
• episode history

Innovation

• Recurrent Memory
• Unified Causal Stream
• Flow-matching Action Head
• Compact Model