Viewing a single comment thread. View all comments

suflaj t1_j3vg5tm wrote

Reply to comment by xenotecc in [D] Have you ever used Knowledge Distillation in practice? by fredlafrite

I think it's a matter of trial and error. The best ratios I've seen were 1:25, but these concerned transformer networks, which are much more sparse than resnets.

There are some tricks, but it depends on the model. Ex. for transformers, it's not just enough to imitate the last layer. I suspect that it's the same for resnets, given they're deep residual networks just like transformers.

1