Actually, hermaphroditism would serve to lower genetic diversity by quickly homogenizing a population. The main point of hermaphroditism in nature is to ensure successful mating in a population which either 1) faces frequent disturbance, 2) has wide dispersal, 3) has a very short lifespan or needs to produce large amounts of offspring (say, because of a high death rate).

In humans and other animals with two distinct sexes, the male is used as a genetic "check": to raise genetic diversity. Otherwise, species become parthenogenic (females producing females without fertilization).