Submitted by fredlafrite t3_106no9h in MachineLearning
There's been a ton of academic work exploring knowledge distillation techniques, sparsity in networks and many others, often with vast numbers of citations. I was wondering what the status of those in real-world ML was. Has any of you used it in a concrete situation? What did you find to work best for you?
Yes, it's the only way to get high throughput high performance models ATM.
With KD and TensorRT you can get close to 100x throughput (compared to eager TF/PyTorch on full model) with 1% performance hit on some models and tasks.
Yes its very useful for text classification tasks. Big transformers get highest accuracy, but we can't deploy them because they are too slow. So we distil knowledge from bigger transformers into smaller transformers or CNNs. If you have a decent amount of unlabeled data to pseudo-label with the teacher, there is barely any loss in accuracy for the student model.
I don't know, but I found this awesome open source site for pruning and fine-tuning various models you may find interesting: https://sparsezoo.neuralmagic.com/
Echoing this, KD is also very useful for taking a heavyweight GPU model and training a student model that's light enough to run on mobile. Small sacrifice in quality for huge performance gains.
Interesting! Echoing this, do you know which kind of companies one could work on this in an applied setting?
Nice! There are 20MB versions of BERT, super interesting thank you!
[removed]
Yes, we used DistilBERT (and even logistic regression) heavily in my previous startup where data volume was web scale.
Depending on the exact problem, large transformer models can be an overkill. For some straightforward text classification even logistic regression with some feature engineering can hit within 3% point of a transformer, and costs a negligible fraction of them.
Neural magic does work in this space, not sure about KD specifically
How small do you make the student, when a teacher is let's say ResNet101? How do you find a good student/teacher size ratio?
Are there any tricks to knowledge distillation? Or just standard vanilla procedure?
I think it's a matter of trial and error. The best ratios I've seen were 1:25, but these concerned transformer networks, which are much more sparse than resnets.
There are some tricks, but it depends on the model. Ex. for transformers, it's not just enough to imitate the last layer. I suspect that it's the same for resnets, given they're deep residual networks just like transformers.
Thank you for the reply!
aaaasd12 t1_j3i3ea8 wrote
It's like transfer learning?
In the company that I'm work only use the normal things like classification tasks/ segmentation with clusters.
Maybe the use case that i see is in NLP with topic modeling using bertopic and tuning the hyperparameters.
But in general simple models are perfect for the tasks that se have.