I honestly have no idea about most people. I bought a small and a medium to see which one fit my purpose better. I wanted it as small as possible since I was spending a few months away from home with just my one backpack. Ended up going with the medium.

Easy to pack, easy to wring out, easy to hang. Can still dry myself off after a swim/shower.

Edit: the one downside compared to a large is that you probably won't be able to drape it over yourself or tie it around your waist.