I started native C++/CUDA. I don't recommend it. Debugging was a nightmare, and the environment was immature at best. There were undocumented exceptions, errors and inconsistencies throughout with unknown dependencies which would break your code catastrophically on a simple driver update.

I pushed into Theano, then TF/Keras and now spend most of my time with PyTorch.

As Nater5000 mentioned, now you can just google any weird error. Someone will have likely encountered it before, and probably has a workaround. For me, I also really appreciate the confidence I can have in my results. If something goes wrong, I can safely assume that I have messed up somewhere. I can have confidene in the outcomes, either good or bad.

If you need something peculiar, then you're probably best off looking at extending or overriding the existing frameworks. For a starter, the frameworks will make your life a lot easier, many examples/videos/articles available for PT/TF. Best of luck with your journey