I have been on just about every SSRI and SNRI on the market. None of them changed my libido much. Some had worse side effects than others, though. And everyone's chemistry is different.
The thing that I can say DID change my sex drive finally was going on birth control. It lowered it- but that was an okay thing because I have PCOS and my testosterone is 3x the normal amount for a woman. It made me a masturbating machine
Now... Worst effects from anti-depressants:
Lexapro: slept 15 hours out of the day, felt like a total zombie. Still masturbated though
I didn't have a partner at the time.
While on Wellbutrin, I went into total hallucination mode and had no clue that other people couldn't hear or see what I did. I had tracers (rainbow trails following my hand if I waved it in front of my face), and I had a cricket that followed me around chirping in my ear. It made me extremely irritable, too.
But, I see that you posted you're taking it on and off... Antidepressants have to build up in your system to actually work well, so you should give it at least a month before quitting it.
And sure all meds have side effects, but I was told by my current neuro-psychiatrist that if I have all of these side effects that means I am not on the right medications.
(My current doc said that I was the worst, mis-medicated case he's worked with. I was at one point on a cocktail of 5 different meds, crying all the time at random for no specific reason, hearing things, and just a total miserable insomniac.)