Might be a bit overkill, but, instead of weight reduction, have you considered changing the architecture completely? I can’t find it so someone maybe can link it below, but, I read an article on using reinforcement learning to decide on an architecture for a neural network. Theirs was designed to just be accurate, however, you could take a similar approach while also adding a parameter to minimize overall size of the network.

Like I said, not really pruning as much as it is about overall size reduction