Submitted by Balance- t3_124eyso in MachineLearning
jrkirby t1_jdzx1ef wrote
Reply to comment by hadaev in [N] OpenAI may have benchmarked GPT-4’s coding ability on it’s own training data by Balance-
I'm guessing the hard part is that you can't "untrain" a model. They hadn't thought "I want to benchmark on these problems later" when they started. Then they spent 20K$+ compute on training. Then they wanted to test it. You can easily find the stuff you want to test on in your training dataset, sure. But you can't so easily remove it and train everything again from scratch.
Thorusss t1_je1z0ib wrote
>Then they spent 20K$+ compute on training.
Your estimate is a few magnitudes too low
AuspiciousApple t1_je2aij3 wrote
Idk, thousands of GPUs going brrrr for months, how much can it cost?
$10?
jrkirby t1_je2f63r wrote
2 million dollars or 20 million dollars is greater than 20 thousand. And it makes the main thesis more salient - the more money you've spent training, the less willing you'll be to retrain the entire model from scratch just to run some benchmarks the "proper" way.
wazis t1_jdzzs1q wrote
Well they can, but it is expensive
Viewing a single comment thread. View all comments