Getting Rid of Inactive Followers

I've been wondering if it's possible to get rid of people who've followed you when they're no longer on the site or banned but are still in our 'people following me' list. It's no big deal, I'm just wondering.