Of the ones I've tried, silicone is the softest, though just how soft it is depends on how it's made and who's making it. If you want something uber-duber incredibly
amazingly squishy and soft, try Tantus's Super Soft formula.
VixSkin silicone is also pretty amazing, and though I only own two VixSkin toys, I think it might be my personal favorite material.