When I installed exhaust ducting in my parents attic I used 6" so the full 100 cfm wouldn't be restricted.

4" is good to use in some instances where firewalls are present. They don't need special protection because they're so small.

If you don't have an easy path to outside from that bathroom, you may need to open up a wall and investigate running an oval or rectangle duct in it to get full airflow.

Otherwise yeah, run the fan at 50 cfm so it's not trying too hard for what the ducting can allow. You could lower the lifespan of the motor, or you're just wasting energy.