Thank you for your reply.

Perhaps I phrased it poorly. You are correct, of course, that increasing model size tends to increase overfitting in the normal sense. Overfitting in this case means a failure of generalization. This would also lead to bad results in new data.

In spoke in the context of this article, which claimed that spurious generalizations are found. LLMs move two parameters up in parallel in order to produced the amazing results that they do. They increase both the quantity of data and the numbers of parameters.