Online Evolution Strategy for Flow-Matching VLA Policies via Self-Supervised Trajectory Distribution Optimization

2026-09-29

AI Summary

This study addresses the challenges of distorted trajectory distributions and the conflation of successful and failed behaviors in Flow Matching-based vision-language-action models. To overcome these limitations, this work proposes an online evolutionary strategy framework that performs exploratory evolution directly within the action trajectory space, leveraging execution feedback to construct a self-supervised objective for model parameter optimization. Furthermore, it incorporates failure experiences as negative feedback regularization and theoretically proves that the resulting objective function constitutes an unbiased estimator of the optimal direction. Notably, this approach achieves policy performance improvements comparable to reinforcement learning fine-tuning in both simulated and real-world environments, without requiring the training of value models or the computation of advantage functions.

Abstract

Vision-Language-Action (VLA) models based on generative frameworks, such as Flow Matching, have recently achieved impressive performance in robotic manipulation. Unlike deterministic policies, Flow Matching enables VLA models to learn conditional action trajectory distributions, where latent noise vectors induce different actions under the same task scenario. However, we observe that these distributions are often ill-formed, with successful and failed behaviors coexisting while considerable probability mass remains in unfavorable regions. To this end, we propose Online-ES, an online adaptation framework for Flow Matching VLAs based on Evolution Strategy (ES), which refines the learned action trajectory distribution through interaction feedback. Instead of pruning the latent noise space, our method performs evolutionary exploration directly in the action trajectory space, where diverse trajectories generated by Flow Matching provide candidate solutions for adaptation. By perturbing sampled trajectories and evaluating their execution outcomes, we derive a self-supervised MSE objective that transfers the evolution direction from trajectory space into model parameter space. Mathematically, we prove that the proposed objective provides an unbiased estimator of the optimal evolution direction. Moreover, we also incorporate failure experiences as negative feedback to regularize the evolution direction, steering the policy away from previously explored failure regions. Experiments in both simulation and real-world environments demonstrate that Online-ES achieves policy improvement comparable to reinforcement fine-tuning, without learning a value model or computing advantages.

Problem

• Vision-Language-Action models
• Flow Matching
• trajectory distribution optimization
• robotic manipulation

Innovation

• Evolution Strategy
• Flow Matching
• Vision-Language-Action
• Self-Supervised Optimization
• Trajectory Distribution