Viewing a single comment thread. View all comments

whata_wonderful_day t1_iv20f1u wrote

Awesome, much appreciate the detailed benchmarks! The dual GPU scaling in particular was of interest to me. I was wondering how the lack of nvlink would affect things.

BERT large benchmarks would also be great, if you could do them?

6

chuanli11 t1_ivhsvnf wrote

BERT large did scales less well for 2x4090. You can find the exact numbers here:
https://github.com/lambdal/deeplearning-benchmark/blob/22.09-py3/pytorch/pytorch-train-throughput-fp32.csv
https://github.com/lambdal/deeplearning-benchmark/blob/22.09-py3/pytorch/pytorch-train-throughput-fp16.csv

2