Viewing a single comment thread. View all comments

TimDarcet t1_j0cpta3 wrote

I think Deit III is pretty sota

2

TimDarcet t1_j0cpy9m wrote

There's also this one with very strong results, but it's a bit less straightforward to train

2

netw0rkf10w OP t1_j0gcgxy wrote

Thanks. DeiT is actually a very nice paper from which one can learn a lot of things. But the training regimes that they used seem a bit long to me: 300 to 800 epochs. The authors of MAE managed to achieve 82.3% for ViT-B after only 100 epochs, so I'm wondering if anyone in the literature has ever been able to match that.

1

TimDarcet t1_j1w6ifs wrote

I think the supervised training they report in MAE is 300 epochs, they used a different recipe compared to finetuning (appendix, page 12, table 11)

2

netw0rkf10w OP t1_j2939o2 wrote

You are right, indeed. Not sure why I missed that. I guess one can conclude that DeiT 3 is currently SoTA for training from scratch.

1