I clean them after each and every use. They get covered in fluids each time and that can harbor bacteria once it's left out to dry. No way am I putting that back in my vagina! Same applies for anal or external toys - I clean them after each use.
For toys I don't use all the time, I'll just let them sit, and I'll rinse them off before using them if they seem linty or dusty. I'm a clean freak when it comes to what goes in my vagina.