No. I think it's perfectly normal. We'll even talk to eachother about who's in our fantasies.

For better or for worse, I usually can't orgasm if I'm thinking of someone other than my wife. However, I think it's perfectly normal to think about other people who you find attractive from time to time.

I've only felt bad about it once and it was because the person in the fantasy was one of his close friends. Just felt awkward.