Everyone can do most stuff from scratch, they don't, because it's pointless to spend time implementing things like ADAM and making sure that it works precisely as it's supposed to when that's not what your research is about or optimizing matrix operations on the GPU, but it's perfectly doable.

People wrote ML stuff in C++ and CUDA as late as 2014. It's still reasonable to make people do things like that as exercises.