It's been used for a millennia so there must be something to it. Just keep in mind a few things:
1. No using it with latex (I'm not sure about silicone, and when in doubt, don't). Olive oil will start the deterioration process.
2. Make sure you're using pure olive oil. You don't want oil that has been infused with a scent or flavor, especially if that flavor is jalapeño!
3. Remember, even virgin olive oil has a natural scent. Some people find it pleasant, others don't. It's a matter of personal preference.
4. Olive oil should be stored at room temperature, away from light, sealed in it's container. It can last up to a year this way though we go through so much of it in my house (for cooking!) that no bottle lasts longer than 60 days.
5. If you do not normally moisturize with olive oil, test it before using it as a lubricant: put a little on the inside of your forearm and wait a few hours (or a day) to see if a rash develops.
Some people say using olive oil can clog pores or promote UTIs, but I've never had that confirmed or denied by a health professional. I know we have a few around here, hopefully one can set the record straight.