The model predicts (in a nutshell) the next word in the answer. It simply can't do math. That's just a known limitation.

I think the argument is quite often "because it works". You throw your data at a standard CNN (if you're dealing with images) and often get decent results quite fast.

What does that even mean? Just the matrix multiplication of a perceptron or including the back propagation of an optimizer?

Isn't that the standard way research papers are written: you only compare your solution to methods that are worse that yours? ;)

From my experience the loss function also plays an important part. Cross entropy forces the model to be very certain with a decision. Focal loss can produce a smoother output distribution.

Or otherwise don't use a pre-trained network for this test. Pytorch randomness shouldn't be better than Tensorflows.

A few random things I learned in practice: "Because it worked" is a valid answer to why you chose certain parameters or algorithms. Older architectures like resnet are still state of the art for certain scenarios. Executive time is crucial, we often take the smallest models available.

And the "results are 0.x% better" papers are often about challenges that aren't interesting anymore since many years.

I think the real answer for many ML problems is "because it works". Why are we using relu (=max(x,0)) instead of sigmoid or tanh as layer activations nowdays? Math would discourage this as the derivative at 0 is not defined, but it's fast an it works.