This is why I both arrive early and wait patiently. Many doctors will schedule more than one appointment at that time and if you're not one of the first people there at that time "slot" then you're going to have to wait. Some doctors might still be out to lunch (depending on your appointment) or some leave by the time you get there.
I have gone to the doctor and it was EMPTY when I got there but a flood of people followed and they all had appointments at the same time. That entire doctor visit took 1.5 hours and it took around 45 minutes to see my doctor (I think).
Don't go on the weekends, make morning appointments (or as early as possible), be early...and be patient. I don't see why people rush doctors so much. I guess I have a higher tolerance for that type of stuff than others. I find other stuff to do like read books, catch up on texting, read emails...I usually don't plan much else in the day if I go to the doctor anyway because I can be there for like 2 hours sometimes.