I don't know where the "soap is bad for mr. limpy" rumor came from (I've heard it too), but my previous packer lasted six *years* thanks to once daily washing with soap and water, and my current one is at a year and seven months with only one tiny pinhole. Antibacterial soap makes it *reek*, but there's a simple solution to that - use regular soap/body wash. I wouldn't leave it in a pool of water indefinitely, and I've heard of packers being ruined by *swimming*, but that's from seawater /chlorine /freshwater bacteria, and long exposure times.
Wash it with a gentle soap or body wash that gets along with your skin, dry it well, and dust it lightly (very lightly!) with cornstarch. Then it won't pick up anything. I've actually had far *fewer* UTIs and yeast infections since I started packing, and I strongly suspect it's because packing makes me pay more attention to what's going on down there, and I've got a daily routine now that, by virtue of keeping my packer clean and non-sticky, keeps *me* cleaner by default.
Keeping it clean doesn't require paranoia levels of washing... just wash your junk when you wash the rest of you, and take the 15 seconds to dust it once it and your hands are dry.