It can be annoying, if you don't do it correctly. What I found helps is to keep the contest running for no longer than a few weeks, so people don't forget about it and so you don't get completely overwhelmed. Be very specific about how people enter- do they comment on your post? Email you? Tweet about it? etc.

I also found it helpful to keep a word document where everyone who entered was added to a numbered list, as they entered. That way I was keeping track of things as they were happening, instead of leaving it until the last minute. I always chose my winners via a random number generator as well, so that way you can just referred to your list of entrants to see which name matches up with that number.

Thanks guys! Very helpful tips. Will post my contest soon!