I'm PRETTY SURE the only reason that whole claim even began is because some vegetarian/vegan was boasting about how much HEALTHIER vegetarians/vegans tend to be in general compared to non-veg people. The healthier you are, the better sex you'll have, since you'll have more of a sex drive, more energy, better mood, stronger muscles (maybe lol), and sometimes more sensitive nerves (some vitamin deficiencies and chemicals in unhealthy food numb or even kill nerves).
However, just because you're vegan or vegetarian doesn't mean you're healthy, ESPECIALLY vegans, you really have to watch your diet and take the right supplements or you'll be in trouble! And things like chips, fries, soda, pastries, and a lot of other junk foods are already vegetarian and/or come in vegan varieties. Oreos are vegan!
You can be totally healthy, if not healthier than a lot of vegetarians/vegans if you still eat meat though, people on specialized diets just tend to be more health conscious.