Playing Atari Space Invaders with Sparse Cosine Optimized Policy Evolution

2025-08-11

AI Summary

To address slow convergence and parameter explosion of evolutionary methods in high-dimensional video game state spaces, this paper proposes Sparse Cosine Optimization-based Evolution (SCOE). SCOE constructs a pseudo-attention mechanism via the Discrete Cosine Transform (DCT), employing energy-guided frequency-domain truncation and sparsification to drastically reduce input dimensionality while preserving semantically critical features. It integrates bilinear affine mapping with the Covariance Matrix Adaptation Evolution Strategy (CMA-ES) for efficient end-to-end policy search. On Atari’s *Space Invaders*, SCOE reduces input dimensionality from 33,600 to 15,625 (53% compression), significantly accelerating convergence and outperforming mainstream baselines—including OpenAI-ES, HyperNEAT, DQN, and A3C—in final policy performance. The core contribution lies in introducing frequency-domain sparse representation into evolutionary strategies, enabling, for the first time, the construction of a semantic-aware, low-dimensional policy space.

Technology Category

Search and Optimization: Learning to SearchMachine Learning: Evolutionary LearningCognitive Modeling & Cognitive Systems: Neural Spike Coding

Application Category

Economics, Online Markets and Human Computation: Cost models of using LLMs in production systemsUser Modeling, Personalization and Recommendation: Attacks and countermeasures in recommendation systemsSearch and Retrieval-Augmented AI: Efficiency and scalability of Web search engines

Abstract

Evolutionary approaches have previously been shown to be effective learning methods for a diverse set of domains. However, the domain of game-playing poses a particular challenge for evolutionary methods due to the inherently large state space of video games. As the size of the input state expands, the size of the policy must also increase in order to effectively learn the temporal patterns in the game space. Consequently, a larger policy must contain more trainable parameters, exponentially increasing the size of the search space. Any increase in search space is highly problematic for evolutionary methods, as increasing the number of trainable parameters is inversely correlated with convergence speed. To reduce the size of the input space while maintaining a meaningful representation of the original space, we introduce Sparse Cosine Optimized Policy Evolution (SCOPE). SCOPE utilizes the Discrete Cosine Transform (DCT) as a pseudo attention mechanism, transforming an input state into a coefficient matrix. By truncating and applying sparsification to this matrix, we reduce the dimensionality of the input space while retaining the highest energy features of the original input. We demonstrate the effectiveness of SCOPE as the policy for the Atari game Space Invaders. In this task, SCOPE with CMA-ES outperforms evolutionary methods that consider an unmodified input state, such as OpenAI-ES and HyperNEAT. SCOPE also outperforms simple reinforcement learning methods, such as DQN and A3C. SCOPE achieves this result through reducing the input size by 53% from 33,600 to 15,625 then using a bilinear affine mapping of sparse DCT coefficients to policy actions learned by the CMA-ES algorithm.

Problem

• Reducing input space dimensionality for evolutionary game learning
• Overcoming large state space challenges in video game policies
• Enhancing convergence speed by optimizing sparse policy parameters

Innovation

• Uses Discrete Cosine Transform for state compression
• Applies sparsification to reduce input dimensionality
• Combines CMA-ES with compressed representation learning