That is a part of it but a very small one compared to other aspects. For the most part is it because languages mix a bit when moving.

The people occupying america before the british got there spoke different languages, and a lot of the people that moved there during the colonies and helped shape the region also spoke other languages. The people growing up in these areas heard different pronunciations and for whatever reason some stuck and some didn't. Rich and poor people do sound different, but that is just part of the local differences, and it is mostly because of the people who surround you rather than anything else.

Australian accent is very rooted in the older british accent because they were a colony, but they also had some mixing and ended up with some different consonants. American accent also rooted in british but also affected by french phonemes and in some areas even phonemes from italian and spanish.

You can also notice scottish and Irish accents have very noticeable phonetics because the original people of those regions spoke languages that used very different phonemes than the rest of western europe, their accents are remnants of their origins.

Regional accents within countries also exist, and are usually traceable to immigration or foreign cultural bubbles. Think new york accent being influenced by italians, or how we use different language words for live animals and their meat.