> RWKV is the exception. When you look at loss against token position, it is comparable with transformers.

Can you link to what you are referring to? If I missed it in the OP post, my apologies.