Viewing a single comment thread. View all comments

bluevase1029 t1_iw1khv8 wrote

Reply to comment by master3243 in [R] ZerO Initialization: Initializing Neural Networks with only Zeros and Ones by hardmaru

I believe it's still difficult to be absolutely certain you have same initialisation across multiple machines, versions of pytorch etc. I could be wrong though.

4

master3243 t1_iw1mpgb wrote

Definitely if each person has a completely different setup.

But that's why we contenirize our setups and use a shared environment setup

2