I think it depends on the kind of oil and on the individual. We had a doctor years ago suggest unrefined organic coconut oil as a sex lube, and we've been using it for that ever since. It's been a blessing on the few occasions when a sex toy has chafed me internally, and I've also used it as part (notice I said part) of a natural yeast infection treatment before since my system didn't tolerate stuff like Monistat and Diflucan well.
However, that same doctor *did* state that most other oils were a no-no, especially petroleum based ones, because they could cause infections.
Like Luscious Lily said, I think the "oil based lubes are bad" thing is just a blanket statement they used in regards to safe sex, because all oils DO affect condom reliability.