Transformers have no spatial/position knowledge unless you add it explicitly (eg positional embeddings). Otherwise they’re a feed forward network with input specific weights.