samloveshummus t1_iw1oyhf wrote on November 12, 2022 at 6:51 AM

Reply to comment by elcric_krej in [R] ZerO Initialization: Initializing Neural Networks with only Zeros and Ones by hardmaru

>I would love if it were picked up as a standard, it seems like the kind of thing that might get rid of a lot of the worst seed hacking out there.

I don't want to be facetious, but what's wrong with "seed hacking"? Maybe that's a fundamental part of making a good model.

If we took someone other than Albert Einstein, and gave them the same education, the same career, the same influences and stresses, would that other person be equally as likely to realise how to explain the photoelectric effect, Brownian motion, blackbody radiation, general relativity and E=mc^(2)? Or was there something special about Einstein's genes meaning we need those initial conditions and that training schedule for it to work.

machinelearner77 t1_iw21k83 wrote on November 12, 2022 at 9:51 AM

I guess the problem with "seed hacking" is just that it reduces trust in the proposed method. People want to build on methods that aren't brittle and if presented model performance depends (too) much on random seed it lowers trust in the method and makes people less likely to want to build on it

samloveshummus t1_iwhwh8y wrote on November 15, 2022 at 7:32 PM

Sure, but maybe it's inescapable.

When we recruit for a job, we first select a candidate from CVs and interviews, and only once we've chosen a candidate do we begin training them.

Do you think it makes sense to strive for a recruitment process that will get perfect results from any candidate, so we can stop wasting time on interviews and just hire whoever? Or is it inevitable that we have to select among candidates before we begin the training? Why should it be different for computers?