It started that way but its becoming more scientific as we understand DNA.

Manatees are closer to elephants than whales but i don't think scientists 150 years ago could have drawn those conclusions. And there are countless examples of reclassification based on new evidence, and to some extent that goes beyond phenotypic expressions.