I LOVE BOOBS! I don't know what it is in a man but they're just awesome. Men are instinctively drawn to larger breast, that's a fact. But honestly I'm not picky. Buuuuut there are some limits. But if a woman is confident with her body then it really doesn't matter. I've been with a girl who had fake full C's and I just wasn't a fan. She always wished they were bigger and it drove me crazy. Now being really flat chested doesn't make a woman, but if they let it interfere with stuff then it just sucks. Be happy with what you got.
The general answer is that it's not a 'primary' attractant for me. But to be honest I've never been attracted to the A-B girls out there. This is observational - I never said to myself 'her boobs are too small'. But the truth of the matter is that the women I've dated have been larger - including my wife of 23+ years. As I tell my kids - what you do speaks so loud that I can't hear what you are saying. Using my own yardstick - it must matter to me - but I couldn't tell you why.
It matters to a certain extent. Personally, I'm not attracted to very large breast sizes. If it's bigger than a large C, I won't initially find that attractive. I also don't like extremely small breasts, like anything less than a large A. I find that middle range incredibly attractive. Hand size, I suppose! However, there are some people who are anomalies, and if I get to know the person and am attracted to them, then I don't care what their cup size is. I won't necessarily pick them out on the street, though (not that I hit on people on the street, but just look their way, I mean).
My college boyfriend once told me that of course he liked my C's but he had always been drawn to smaller breasts too. lol. I think that on the whole, it matters, but everyone really can be a bit different.