You're correct, it has to do with acceleration. This question has a name, the Twin Paradox and solving this requires a more complex application of relativity than is normally taught until grad school- but essentially, it all comes down the the acceleration.