Keep everything exactly the same when you try both, i.e. just unplug the HDMI form the back of the PS5 and plug in straight into your Xbox. This way you're on the same port/input and using the same cable. Also make sure the Xbox is set to output in 4K and HDR and all that. There should not be a noticeable difference so something is up.