You're confusing sex with characteristics associated with sexual dimorphism. The only two categories involved in defining a person's sex are chromosomal and gonadal, and ultimately gonadal wins out in any case where the two can't otherwise be reconciled; what really matters for most medical and social purposes is a person's phenotypical, or gonadal, sex.

None of the genetic disorders resulting in "intersex disorders" creates a situation in which the functional sex of a living person is in any real question. The most compelling, case, obviously, would be weak or missing SRY, but we already have words for making distinctions between displayed characteristics and people's genotypes: phenotypes.

People with intersex disorders still have a definitive medical sex, albeit there are obviously other complications concerning what medicines will or won't kill you in cases where the disorder drastically affects your endocrinology. There's no "somewhere in between", there isn't some magical incremental slider between male and female.