A number of factors here! Chlorine and saltwater are huge contributors, and swimming tends to involve a lot of sun exposure, plus there's tangling and agitation.

More specifically, agitation of any animal hair fiber while wet means the cuticle scales have been softened by the water, so they've lifted up a little, and each strand is rubbing with its neighbors, scraping and damaging each other's lifted scales.

Incidentally, this mechanical agitation of strands against one another while they have lifted and softened cuticle scales is also how woolen clothing ends up felted if you throw it in the washing machine.