I have yet to see it "damage" a toy so it cannot be used. It can adhear to the silicone material, which makes it sort of gummy feeling. Most of the time it goes back to normal after a few washing's. The big problem, at least to me, is that I would not like that feeling during use.
For some more "Lube testing" you can check out my workshop
on lube compatibility.