Viewing a single comment thread. View all comments

All-DayErrDay t1_j3xttzb wrote

500k, actually (per MosaicML). Will likely drop to 100k soon with H100s being several times faster. Would probably be even lower if you added every efficiency gain currently available.

2

m98789 t1_j3xxyvm wrote

You are right that the trend is for costs to go down. It was originally reported that it took $12M in compute costs for a single training run of GPT-3 (source).

H100s will make a significant difference and all the optimization techniques. So I agree prices will drop a lot, but for the foreseeable future, still be out of reach for mere mortals.

2