Start with fingers. They're sensitive, small, convienient, and usually inexpensive (unless you manage to give yourself carpal tunnel syndrome from your play). A glove helps to smooth things out and keep things clean for little additional cost, but do use lube.

From there the next step is silicone. Silicone lasts forever and is clean, safe and flexible - a good combination. If you can only play with one finger get a one-inch toy, two fingers can usually go up to 1-1/2. Look for a probe, dildo or bead stick if you like thrusting, a plug if you like the feeling of something there without having to worry about holding it in.

From there you'll start to find out what you like. You can branch out in size, style, and materials (glass and steel can be very nice).