My husband and I are non-monogamous. I call myself a polyamorist, and I would call my husband that as well, although left to his own devices he would just be monogamous.
"Swingers" seems to be a classification that poly folks almost "pooh-pooh", at least in my local community. The difference between the two is one of definitions and extremes. At the exteremes, Someone who sport-f*cks is a swinger, someone who brings extra people into their life and family is a polyamorist. Unfortunately, however, most people fall into the spectrum in between, or move around as to where they stand, which can make it hard to exactly state the difference between the two.
I kind of like to think of swinging as a subset of poly. Polyamory is the ability and inclination to form meaningful intimate relationships with many people. That can look like a lot of things, including swinging. Swingers like to have sex with other people as a passtime - but they can certainly form meaningful and intimate relationships too. Swingers seem to be generally hetero couples, but perhaps that's changing.
Interested to hear other thoughts.