I've never heard of an actual study that proved that sex toys can do permanent damage. While it IS possible, it's generally outside the realm of extreme possibility.
As LikeSunshineDust said, you'd have to do actual nerve damage to desensitize that part of your body. There are toys out there that can lead to temporary numbness, however, and also toys that can cause pain.
Pain, you can get over and heal from. Numbness is temporary.
For instance, the Hitachi Magic Wand is *enormously* popular. It has a very, very high success rate as a clitoral stimulator. Unfortunately, I'm in the less than 1% out there that doesn't like it at all. It has a tendancy to numb me out before pleasure is even attained. It's a remarkaby powerful vibrator, so it overloads my senses and boom! That's it. Nothing else happens. I got over it and moved on, and my body was the same after some belated soreness.
Don't sweat it. It's a very popular rumor for people who don't use sex toys. I've heard it myself, as well, and even feared it all by myself. But truth? I own well over 25 sex toys, and many of them vibrate. I'm not desensitized from them, far from it, readily available pleasure has a tendancy to make me *more* sensitive, and *more* receptive.