The Y chromosome has surprisingly little information/genes in it. Basically it contains one gene that activates all of the male-only genes on other chromosomes. So no, maternal chromosomes/genes contribute to male genital formation too and half of your father’s chromosomes/genes (that affect genital formation) will not be inherited by you