Boxer briefs, here. The packer tends to migrate too much for me in boxers. I used briefs for a while before I got boxer briefs and never had a problem with it rubbing, looking weird or irritating me.
In most cases, the reason a packer causes irritation when packed against the skin is because it's not being kept *clean*... a daily wash with soap and light dusting of cornstarch goes a *long* way. Cis guys wash their junk, and trans* and genderqueer folks shouldn't forget to wash theirs!
Of course, if you have an allergy to plastics or your anatomy is positioned such that it's rubbing *directly* against the packer, that's a different story, but in general, if you're packing against the skin, keep it clean and you'll be fine.