One issue with aluminum is that it is very chemically reactive. It produces toxic fumes if it comes in contact with a base (alkali) and can react with acids also. I just read recently about a kid being kicked out of school for exploding aluminum foil and drain cleaner.
I don't know about the sex toy issues or the properties of whatever is used to color it.
I know that some people are concerned that aluminum is linked to Alzheimer's and brain cancer, though whether or not that has anything to do with the safety rating is purely speculation. From what I recall reading, though, aluminum toxicity is mainly a concern when you're inhaling it or if you have severe kidney disease. I mean, I use it pretty frequently in cooking and it's even in deodorant and stuff (which is why people actually made a huge stink about it to begin with).
My guess is that because they are anodized this means that there is a chance that the coating could become scratched. Once the coating is scratched it looses its non-porous state. Whereas with silicone and stainless the non-porous nature stays no matter what you do to it.