For vaginal exercise (Kegels)? From my understanding and experience, there's no risk. I use a stone Kegel egg that doesn't have a removal cord and it's kind of large at 5 inches circumference, so I have to 'bear down' on my muscles to expel it (I find a squatting position works well), but it has never felt stuck. It's not like anything can get lost up there, so as long as you relax when you want to remove the balls, it shouldn't be too challenging to get them out.
I think the material also makes a difference; I gave up on silicone exercisers because they seemed to suck up what little natural lubrication I have, and removal was uncomfortable. On the other hand, my stone egg is very smooth, which makes insertion and removal a breeze. Glass and metal balls would be slick, so they'd be easier to insert and remove as well.
Hope that helped!