That’s a great paper! I like this paper too that looks at ViT’s through a signal processing lens and points out some potential flaws in the architecture for vision applications

there is a version on Replicate you can try easily

https://replicate.com/methexis-inc/img2prompt

you can do n-digit addition of positive integers as a sequence where each digit is a token, i.e.

the problem 946 + 82 = 1028 could be made into sequence of:

9 | 4 | 6 | + | 0 | 8 | 2 | = | 1 | 0 | 2 | 8

(you could also omit + and = tokens).

Andrej Karpathy uses this task in his minGPT repo.

edit: also in that repo he does character level training on a tiny dataset of Shakespeare writing