Viewing a single comment thread. View all comments

pr0d_ t1_izqjmmk wrote

Reply to comment by MazenAmria in Advices for Deep Learning Research on SWIN Transformer and Knowledge Distillation by MazenAmria

yeah as per my comment, the DEiT papers explored knowledge distillation based off Vision Transformers. What you want to do here is probably similar, and the resources needed to prove it is huge to say the list. Any chance you've discussed this with your advisor?

1

MazenAmria OP t1_izrgnco wrote

I remember reading it, I'll read it again and discuss it. Thanks.

1