Sorry, dictionary issue. I meant Self Attention (I ve edited previous answer)

I would scale logits before softmax, like it’s been done in self attention.Actually that scaling in self attn is to make the final dist of the attention weights to be smooth.