Viewing a single comment thread. View all comments

visarga t1_ivioifb wrote

Reply to comment by eliyah23rd in The big data delusion – the more data we have, the harder it is to find meaningful patterns in the world. by IAI_Admin

> They overcome overfitting using hundreds of billions of parameters

Increasing model size usually increases overfitting. The opposite effect comes from increasing the dataset size.

6

eliyah23rd t1_ivjri3f wrote

Thank you for your reply.

Perhaps I phrased it poorly. You are correct, of course, that increasing model size tends to increase overfitting in the normal sense. Overfitting in this case means a failure of generalization. This would also lead to bad results in new data.

In spoke in the context of this article, which claimed that spurious generalizations are found. LLMs move two parameters up in parallel in order to produced the amazing results that they do. They increase both the quantity of data and the numbers of parameters.

1