Evolution strategies as a scalable alternative to reinforcement learning

OpenAI recently published a blog post evaluating evolution strategies (ES) as a competitive alternative to reinforcement learning (RL) for training AI agents. ES is a decades-old black-box optimization technique that was largely overshadowed by gradient-based methods. The researchers found that ES matches the performance of standard RL algorithms on popular benchmarks like Atari games and MuJoCo robotics tasks, while offering practical advantages: it scales easily across thousands of parallel workers, does not require backpropagation through time, and is robust to sparse or delayed reward signals. This makes ES particularly appealing for developers building AI workflows in distributed computing environments. However, the authors note that ES tends to be less sample efficient in terms of total interactions, requiring more environment steps to achieve comparable results. For builders, the key insight is that ES can serve as a drop-in replacement when RL is impractical due to long training times or complex credit assignment. The simplicity of the algorithm also makes it easier to implement and debug without specialized RL libraries. OpenAI's work reinforces the value of revisiting older optimization techniques with modern hardware.