A couple of questions:
Are you the one holding your smaller toys, or do you let him hold them? If it's you, then you should bring him into the toy play to get used to someone else stimulating you there.
Definitely get a toy that's closer to his size. In fact, if there's a big size difference get a couple to step up. Go nice and slow with plenty of foreplay, and only move up when you're confident you're ready.
Third, pay attention to position. If a position is comfortable with toys it's more likely to be comfortable with a penis, if it's not comfortable with either try something else.