It depends on what the scandal is. If the scandal is that they cheated on their spouse, I would lose respect for them, and may or may not vote for them depending on the rest of their positions. If the scandal was that they lived an alternate sexual lifestyle (into BDSM, swinging, cross-dressing, same-sex, whatever), it would not affect my vote at all. If the scandal is some sort of sexual misconduct (exposing themselves, using their position of power to demand sex, being inappropriate with a minor, assault, rape), then I would never vote for them, no matter what my feelings on their other positions.
Though I have to say, I think the stupidest sex scandal we've had in a while is Jack Ryan - he was accused of asking his wife if she wanted to have sex, and taking no for an answer. The location was unorthodox, but I don't care if a politician attends sex clubs, as long as they are following the rules of the club.