Genitals have bacteria on and in them, and the anus and mouth has even more. (We have a lot of silicone anal toys, and my husband likes me to play with phallic vaginal toys orally.) But, even if I only used toys vaginally, I still feel that that bacteria has to be washed off with hot running water and soap after every use. Sometimes we wait until morning (in fact, I didn't sleep well last night, and was very sleepy early this morning, but couldn't sleep until I cleaned the 3 toys we used during our play time last night.)
Silicone may not grow bacteria the way jelly or rubber does, but any bacteria on the toy can cause potential problems if reused before washing.
It takes a few minutes, and I wouldn't take the chance by not washing them. I use wipes, but I always wash them in soap and water later. Wipes and toy wash leave chemicals on your toys that I prefer not to have in my body.
Think of it this way: would you eat dinner on a plate and use a fork and knife and then just put them back in the cabinet without washing for an other use "because they didn't look dirty?" I know I wouldn't.