Evolving Cache Schedules for Fast Diffusion Policy Inference

2026-07-22

AI Summary

This work addresses the high computational overhead of diffusion policies in real-time deployment, which stems from repeated denoising steps, and the inefficacy of existing caching strategies that uniformly allocate resources, thereby failing to balance efficiency and performance. To overcome these limitations, the authors propose EVO, a training-free acceleration framework that leverages evolutionary search to globally optimize cache refresh scheduling over a block–timestep grid, thereby skipping redundant computations. EVO integrates redundancy-aware initialization and a goal-conditioned early-stopping mechanism, enabling highly efficient inference without fine-tuning the pretrained model. Experimental results demonstrate that EVO achieves up to an 8.05× speedup in action generation and reduces FLOPs from 15.77G to 1.96G while maintaining near-original task performance across multiple manipulation benchmarks.

Abstract

Diffusion policies achieve strong visuomotor control by iteratively denoising action chunks, but repeated denoising makes real-time deployment computationally demanding. Cache-based methods reduce inference cost by reusing intermediate activations, but existing training-free schedules typically allocate computation uniformly across blocks, ignoring heterogeneous redundancy across blocks and leading to a suboptimal performance-efficiency trade-off. To bridge this gap, we introduce Evolving Cache Schedules (EVO), a training-free acceleration framework that globally schedules cache refreshes via evolutionary search. EVO represents each candidate as a complete schedule over the block-timestep lattice. Thus, redundant transformer computations during iterative denoising can be skipped through cache reuse while preserving closed-loop rollout performance. To make the search practical, EVO introduces redundancy-aware initialization, which seeds the population with promising schedules, and target-conditioned early stopping, which verifies and terminates once a desired performance target is reached. The offline-optimized schedule can be directly plugged into pretrained diffusion policies without retraining. Extensive manipulation benchmarks show that EVO preserves near-full performance while substantially reducing computation, achieving up to 8.05x action-generation speedup and reducing FLOPs from 15.77G to as low as 1.96G. Source code is available at https://github.com/pillom/EVO.

Problem

• diffusion policies
• cache schedules
• inference acceleration
• computation redundancy
• real-time deployment

Innovation

• diffusion policy
• cache scheduling
• evolutionary search
• training-free acceleration
• redundancy-aware optimization