IMHO it's not the vibe, although some materials require adding lube, it's that you stop lubricating. I might be totally wrong but it works (or rather stops working) the same for me. I'm a special case - a fetishist. I need specific fantasies to maintain arousal - otherwise even my Amethyst won't stop me from drying up. It might be that using the toy brings you out of your game - it disrupts the arousal you built up in your head.
Whatever it is, I second the suggestion for good lube. Silicone, if your toy can take it.