No, and it’s exactly because of what you suspected: we are operating in a too small area for the expansion of the universe to affect space travel much at all.

Here’s why: the gravity exerted between the galaxies within our Local Galactic Group is stronger than the Dark Energy pushing the universe apart. The LGC is 10 million light years in diameter while our closest star is 4.25 light years. If the expansion of the universe doesn’t affect galaxies within millions of light years from us, and we’re unlikely to even travel 4.25 light years away, this expansion doesn’t matter when considering human space travel.

For more context, our Local Galactic Group is a region of 30 major and hundreds of minor galaxies, of which the Milky Way is the second largest, and it extends for 10 million light years.

And regarding human space travel: We won’t even go to the nearest solar system anytime soon because the closest star to us, Proxima Centauri, is 4.25 light years away. The Voyager 1 would take 73,000 to reach it going 61,500 km/h and the fastest manned space flight was Apollo 10 at just under 40,000 km/h.