You should be ok since it was only twice, but in the future, I recommend always using a condom over jelly/sil-a-gel toys. They are porous, so they harbor bacteria. It's especially important to use a condom if sharing the toy with others, as this type of material cannot be properly cleaned and therefore STIs can be spread.

Phthlates are also a potential health concern. Cheaper, lower quality toys often contain them. A strong, plasticy odor is often indicative of phthlates. Do some research and make up your mind about whether or not you want to put yourself in contact with them.

If you'd prefer a safer material, go with silicone next time. It's much higher quality and it will last you a lot longer.

Here is the materials page, where you can learn about the safety of various sex toy materials.