Here's 'Language modelling with pixels'! https://openreview.net/pdf?id=FkSp8VW8RjH It gets close to BERT in English peformance. It does better in other languages but that's probably only cause BERT wasn't trained much on them afaik. but still! its apparently much more viable thought