From PZ Myers:
...the gene for the androgen receptor, a protein absolutely essential to the development of his masculinity, is located on the X chromosome—you know, that female thing.As men, we can only get our X chromosome from our mothers. That same X chromosome is also crucial to our masculinity in the most fundamental sense.
(I suspect the explanation is rather mundane - the X chromosome, if I recall my high school biology, is much larger and more complex than the Y.)
Still, the idea that it's only through our mothers' genetic gifts that we can be Manly Men makes me smile.