So, your fan isn't coming with a reducer like I was thinking. It comes with an adapter to go from a 4" oval to 3" round duct. The area of the both sides is probably very similar but without knowing the width or height of the oval, I can't be certain. This fan is designed to work with this adapter.

Your air blowback could be due to duct blockage or otherwise reduced throughput. Typically, there are maximum duct lengths and number of elbow (bends) allowed before throughput is reduced. Without putting eyes on your ducting, you've really no way to know. I don't know if HVAC companies will scope your duct or not, but I'm fairly certain they can clean them.