Viewing a single comment thread. View all comments

DeepNonseNse t1_iqvgzsk wrote

0

Desperate-Whereas50 t1_iqwzlgc wrote

I am not a transformer expert. So maybe this is a stupid question, but is this also true for transformer based architectures? For example BERT uses 12/24 transformer Blocks. Thats sounds not as deep as for example a resnet-256.

1

ResourceResearch t1_iro8zof wrote

Afaik it is not clear. In my personal experience, the number of parameters is more important, rather then the layer size, i.e. a smaller number of wider layers does the same job as a large number of narrower layers.

Consider this paper for empirical insights for large models: https://arxiv.org/pdf/2001.08361.pdf

1