There is a frame in which everything is in the Hubble follow, basically moving with the redshift one would expect from the expansion of the universe. But within this flow you can independently move (e.g. you can get in a car and drive any direction). This will then deviate you from this flow. Though if not acted upon by another force, you'll eventually be "dragged" into this flow since the universe continues to expand in all directions.

In other words spacetime is kinda fucky