Viewing a single comment thread. View all comments

jrkirby t1_jdzx1ef wrote

I'm guessing the hard part is that you can't "untrain" a model. They hadn't thought "I want to benchmark on these problems later" when they started. Then they spent 20K$+ compute on training. Then they wanted to test it. You can easily find the stuff you want to test on in your training dataset, sure. But you can't so easily remove it and train everything again from scratch.

7

Thorusss t1_je1z0ib wrote

>Then they spent 20K$+ compute on training.

Your estimate is a few magnitudes too low

9

AuspiciousApple t1_je2aij3 wrote

Idk, thousands of GPUs going brrrr for months, how much can it cost?

$10?

2

jrkirby t1_je2f63r wrote

2 million dollars or 20 million dollars is greater than 20 thousand. And it makes the main thesis more salient - the more money you've spent training, the less willing you'll be to retrain the entire model from scratch just to run some benchmarks the "proper" way.

1

wazis t1_jdzzs1q wrote

Well they can, but it is expensive

3