Great to hear! I actually lead the CMA-ES effort and tried very hard to match the fine details of pycma so that the performance is comparable. If you run into any unexpected behavior please do open a Github issue or reach out to me directly. There's a lot of fine details in practical CMA-ES implementation, so I'd really like to know if I missed anything.

It depends what you're trying to do :)

If you want to run GPU-accelerated neuroevolution in Brax or IsaacGym, then keeping everything on GPU is absolutely relevant. Similarly if you're trying to do MPC or any optimisation of an NN input, then its still very useful to be on the GPU. As you said, bench-marking is another place this GPU acceleration can be very helpful. Basically anywhere where the fitness evaluation isn't the only bounding factor.

For expensive/CPU-bounded fitness functions, we have other utilities too! For example, with a single flag you can distribute your fitness evaluation across multiple actors using ray. This means you can scale to an entire CPU cluster effortlessly!