Viewing a single comment thread. View all comments

limpbizkit4prez t1_jahaq8v wrote

The authors kept increasing model size until the model overfit the task. I'm not sure if that's high impact. It's cool and everything, but over fitting a data set is never really valuable.

4

MysteryInc152 OP t1_jahgb2n wrote

Overfitting comes the necessary connotation that the model does not generalize well to instances of the task outside the training data.

As long as what the model creates is novel and works, "overfitting" seems like an unimportant if not misleading distinction.

2

limpbizkit4prez t1_jahhmhd wrote

Lol, I strongly disagree. There are already methods out there that provide architecture design. This is a "that's neat" type of project, but I'd be really disappointed to see this anywhere other than arxiv.

3

_Arsenie_Boca_ t1_jai5zgz wrote

The final evaluation is done on test metrics right? If so, why does it matter?

2

limpbizkit4prez t1_jai7l96 wrote

It matters because the authors continue to increase model capacity to do better on a single task and that's it. They also determined that strategy, not the LLM. It would be way cooler if they constrained the problem to roughly the same number of parameters and showed generalization across multiple tasks. Again, it's neat, just not innovative or sexy.

2