Oh man, I have SO been there. And like you said, I don't think it's a rare thing for women. I know for me it was hard because my partner can get off from a stiff breeze with zero foreplay. But she has the same insecurities that she gets off too quickly, so I do think it's an insecurity thing in general.
I struggled with this for a long time and my anxiety definitely ruined the mood of sexy-time more than once. The thing that finally helped me overcome this was talking it out. A lot. I communicated my anxieties to her and found out that it didn't bother her at all. I figured she must be frustrated and bored after a while, but my fiancee was great--super patient and actually loved every minute. Mind you, it took me a while to actually believe that she wasn't just humoring me. We talked it over several times and I was eventually convinced.
What also helped is that she promised that if she ever felt like I was just taking too long for her hands/mouth/etc to handle, that she would tell me. I made her promise--that way I wasn't lying there wondering if she was really enjoying herself at all. And you know what? She's never told me she wanted to stop.
That confidence in trusting and believing in her to tell me if she needs to adjust (which she has and no worries) or if she is getting impatient, really allows me to relax and enjoy myself without worrying about what's going on in her head. So it can be overcome and you'll be surprised how much quicker you can come once you stop worrying oh-god-s/he's-got-to-b e-hoping-it's-any-seco nd-now. Because if they love you--they're likely not thinking that. And if they are: you've got other things to think over.
Best of luck!