Latex allergies are often pretty serious. I wouldn't risk it - get something with 100% known and safe ingredients. 100% silicone, glass, metal, etc.
You could write Doc Johnson's directly (or whoever makes other toys you're interested in) and ask if they'll guarantee it's safe for people with latex allergies. If they won't guarantee it I definitely wouldn't risk it.