Oh my God. I used to do this too! I am happy I am not the only one!! But my monkey brain learned not to do this eventually. I have managed to get it to GRU performance by applying more warmup steps, learning rate scheduling, decreasing model size, using Pre-LN, doubling the batch size, and reducing the sequence length.