Well, that's because we have an investment in a particular present, but not a particular future. So what is considered a "massive" change to the present would not be an inconsequential change to the future.

If you went back 2000 years, the tiniest actions could have massive consequences. You could end up delaying someone by 5 minutes, and as such they get home 5 minutes late, and when they have a romantic encounter with their partner, a different sperm ends up fertilizing the egg, so a different child is born, and that person is likely an ancestor of ~20% of all people alive today, so even if nothing else changes, the entire world is now filled with different people. Each of those people will have done slightly different things, meaning they have similar effects on other conception events, and now nobody who exists in your history books exists any more. The world you came from is gone, and all it took was a 5 minute delay.

But, we don't relish the power of minor inconveniencing others, because from our perspective, while a 5 minute delay might have the same level of effect 2000 years from now, no particular future is seen as the "current" one that must be preserved at all costs. And we can't necessarily influence the future in a way we want, or even in a way that hold any meaning to us.