MEMENTO: Memory-Guided Memetic Code-as-Policy Evolution

2026-07-24

AI Summary

This work addresses the challenge of long-horizon embodied tasks, where policies must execute extended sequences of interdependent actions to obtain sparse rewards, and existing evolutionary methods lack effective mechanisms for local sequence optimization. The authors propose the first memory-guided single-elitist membrane algorithm framework, representing policies as executable programs and integrating structured feedback, multi-branch search, macro-mutation, and crossover operators. Crucially, the approach incorporates large language model–based code generation and a memory-guided local hill-climbing strategy. Evaluated on Robosuite and AI2-THOR, the method significantly outperforms Eureka and REvolve, demonstrating strong generalization to unseen environments and successfully transferring policies from simulation to a real-world Franka robot.

Abstract

Long-horizon embodied tasks require policies that execute many dependent actions before task success can be observed. Representing policies as executable control pro- grams (code-as-policy) enables their decision logic to be inspected and revised after rollout evaluation. Revised programs can then be executed and compared by rollout performance, framing policy improvement as execution-guided program search. Evo- lutionary methods driven by large language models (LLMs) provide a natural mecha- nism for this search by generating variants and selecting high-performing candidates. However, existing approaches primarily select among independently generated vari- ants and lack a sequential local improvement phase. We introduce MEMENTO, a memory-guided single-elite memetic framework for code-as-policy evolution. ME- MENTO first evolves a rollout evaluator that maps policy rollouts to scalar fitness and structured feedback metrics. Fitness selects accepted candidates and the next elite, while feedback metrics condition policy proposals generated by memory-guided hill-climbing, macro-mutation, and crossover. We evaluate MEMENTO on two long- horizon embodied domains: Robosuite Franka Tower-of-Hanoi manipulation and AI2- THOR household interaction. MEMENTO outperforms Eureka and REvolve, adapted as code-as-policy evolutionary baselines, in task success and generalization to held- out Robosuite object configurations and unseen AI2-THOR scenes. Ablations show that zero-shot generation and unevolved evaluators fail to solve either domain, and that removing policy-search branches reduces performance. Finally, we deploy the best-evolved Robosuite policy on a physical Franka robot, demonstrating the feasibil- ity of sim-to-real transfer of the evolved code-as-policy. Code, prompts, and videos are available at: https://github.com/sygkounas/MEMENTO.

Problem

• long-horizon embodied tasks
• code-as-policy
• evolutionary optimization
• policy search
• memetic algorithms

Innovation

• code-as-policy
• memetic evolution
• memory-guided search
• embodied AI
• LLM-driven program synthesis