It is probably your personal preference. Glass and silicone are my favorite. Both are easy to clean, and non-porous. Glass is weighty too, and lends itself easily to temperature play. I like silicone toys with a velvety texture, but one downside is that sometimes toys will have a seam from the mold.
All of those are good, depending on your personal choices. Do you like stiff toys or not? Do you like the feeling of a "heavy" toy? A toy that holds temperatures for a long time? These affect your choices.
I always recommend that you start with silicone. Silicone bends so you're less likely to injure yourself, it's easy to clean and safe, moderately priced, and easy to take care of.
I'm an equal fan of both silicone and glass. The have different functions in my use and each brings something special to the table. And of course, each is durable, safe, and easy to clean. I can't speak for metals [yet] but I'm pretty sure those are nice too, especially if you like glass. The hard materials are great for applying pressure in a specific spot.