no one ever remembers john vitter or john ensign. vitter visited prostitutes, is still a senator and was re-elected in 2010 by a landslide.
While a scandal doesn't always mean a person is unfit for office, i try to consider 3 things:
1. If they did this to their significant other, is it likely they'd do it to me (screw over/lie).
2. if they're that stupid, should they be in office (weiner texting a pic of his weiner for example).
3. was it a crime (for vitter, yes, because prostitution is illegal)
also, here's an snl skit poking fun at no one knowing about vitter or ensign.