Depends on what I'm using it for. I have soft silicone, hard silicone, glass and steel "dildo" type toys (some are "crossovers" with massagers/probes/whate ver). Glass and metal are stiff, slick, and good with heat (or cool), but they aren't so good for wild abandon play. Silicone is nice for that or if I want something to "clench" around and feel squeeze.
Definitely start with silicone (if you're building your collection), as it's more forgiving.