Eh, I just wouldn't do it. I mean it only takes a few minutes to rub some soap over it and wash it. If cleaning is that much of an inconvenience I say grab some free wipes with all of your orders or go buy some from the store to keep on hand, it only takes some seconds to swipe with the wipe a few times.
I'd be worried that it could cause an infection down there. It just doesn't seem very sanitary. As someone that is prone to and gets vaginal infections easily I am very paranoid about keeping my stuff clean and wouldn't take the risk no matter how tired or lazy I'm feeling. Almost all of my toys are glass or 100% silicone and I'm the only one that uses them but I'm still obsessive about cleaning.