I have an XS and my friend has a S, the S is CONSIDERABLY larger, but, the XS has extra large balls which prop the penis up more, giving you more of an outward bulge than the S. I like my XS, but I tend to tuck the balls between my legs a bit to avoid the boner look.
Most guys go for the extra small because it's less "inappropriate". However, I don't think height has much correlation with penis size :S everyone seems to think so. Pornstar Ron Jeremy was your height and was well known for his 10 inch (approx.) penis. I think weight has more of a relation to penis size (skinnier guys seem to be more hung), but that's just my own observations xD.