This is not a privacy problem based on how Apple has architected the data. From the article:

That means that the location info is stored on your iPhone, and is encrypted and dissociated from you if it needs to leave the device.

No outsiders can see your Significant Locations, and they're not accessible through iCloud either.