I've never heard any guys say anything against it. Maybe try washing it after you're done swimming? that might help so that the chlorine isn't staying on it. As far as pond/lake/ocean water goes, you shouldn't worry. Although, if you swim in the ocean, you should definitely wash it after because of the salt. I hear more guys warning about going swimming in a chlorinated pool with your binder than with your packer.