Before and after every use, although sometimes the before is just toy cleaner/wipes and the after might be in two stages -- one quickie cleaning immediately after use, and then a thorough one when I'm sure not to be caught on the way to and from the bathroom.
I only use glass, metal, and pure silicone on any sort of a regular basis, though. My TPR toys are only used when there's absolutely no chance of anyone coming home early, because those need to be scrubbed before and after, plus they're noisy anyway.
Every single item I own has its own lint-free storage bag, and the TPR toys are in Sugar Saks. I don't have anything jelly that I actually use.