If focal loss is interesting, check out polyloss, which is a generalization of the focal loss idea.

Have you tried something like this?

You're not able to overfit on the hard examples alone? Why not?

The alternative is writing your own cuda code or C++. Fortunately for you pytorch is pretty easily extendable. If you have something that needs to be done quickly, why not write a cpp extension?

If you're using pytorch, what's stopping you from using the C++ api? Seems like it is exactly what you are asking for.

>So if we ignore the implementation details to accomplish this for large networks, are there any inherent advantages to using higher-order neurons?

I don't know what that might be, but there is an inherent advantage in stacking layers of act(WX+b) where act is some non-linear function. Instead of guessing what higher level function you should use for each neuron, you can learn the higher order function by stacking many simpler non-linear functions. That way the solution is general and can work over many different datasets and modalities.