To add to it, non random weight might be worse for tiny/simple models.

But modern vision models should be fine with it.

Like bert text weights is a good starting point for images classification.