For what it’s worth, I believe the significant locations are not provide to Apple and are only available directly in your device, so without access to the unlocked phone, they are private.

Someone can correct me if I’m wrong on that.

But if you want to disable, this article explains how.

You can see my example here:

https://imgur.com/nUQEMtC

While there are 57 records, you can only see the last record (Flea Street Cafe for a yummy dinner with my wife).

You can turn the setting off and/or clear history:

Settings->Privacy&Security->Location Services->System Services (scroll to bottom) ->Significant Locations (again at bottom)

It's also worth noting that you need Face ID or passcode to not only unlock your iPhone, but again to get to that screen. So if you handed someone your unlocked iPhone, they wouldn't be able to see your last record unless they did a second unlock at that page.