Hello, does anyone have experience with vision transformers?

I get wierd grid artifacts, especially on white / bright, textureless walls or sky.

Here is how it looks like: https://imgur.com/a/dwF69Z3
Im using maxim architecture: https://github.com/vztu/maxim-pytorch

My general task is image enchancement (make image prettier)
I have also tried simple GAN methods https://github.com/eezkni/UEGAN which doesnt have such issues

I have researched a bit but im unable to formualte this problem properly. I have found that guided filters might help here but havent tested them yet. Thanks