Some people feel that it's an act of violence against women -- since it can be painful if not done right, there are plenty of folks out there that assume it just is painful. Also, there's the portrayal of it in porn that's not doing it any favors.
Other people are homophobic for whatever reason, and are under the impression that it's just for homosexuals -- there are men out there that fear enjoying prostate stimulation automatically means they're gay and just haven't realized it.
Then there's the embarrassment factor, since it's really easy to not feel clean about anything in that region, and therefore be afraid to try it for that type of reason.
Finally, there's the unfortunate label of "Sodomy" which has been tied to the Biblical story of Sodom. Many Christians truly believe that the central sin in that story was the sex act, but if you truly read and study the Bible, you discover that God's beef with Sodom wasn't immoral sex, it was greed, indifference toward the suffering of others, and a general lack of charity. The Sodomites weren't slutty, they were selfish. That's why Lot, the charitable man who offered hospitality to strangers, was the one spared.