If the force that accelerated you were constant, you could indeed reach a speed arbitrarily close to the speed of light just by starting arbitrarily far away – and escape velocity wouldn't be a thing, either.

But gravity is inversely proportional to distance squared, so if you're very far away (in a toy universe where there are only you plus the object you're falling towards), your initial acceleration will also be very slow and almost all of the speed gain will happen when you're already very close to the object.