Yep, it's concerning for sure. Oh, and here are just a few (of many) reasons why "I'm not doing anything illegal, so why should I care?" completely misses the point:
(1) Taken to its logical conclusion, that attitude is patently absurd. Let's assume for the moment that you actually don't do anything wrong or illegal in your day to day life. Are you REALLY only concerned with surveillance if there's a chance you might get caught for something? Hey, since good ol' Gdom here doesn't have any law enforcement power, I presume you'd be alright with me just putting a few cameras up in your house, right? But you're probably not alright with that, correct? That would be super weird and creepy. (And you probably aren't even alright with me collecting your metadata) You might trust the government more than someone like me. That makes sense to a degree, but it seems you'd have to believe that the government is all-beneficent to give them that much trust (call me cynical, but that strikes me as profoundly naive).
(2) The fact that you have, in fact, done nothing illegal doesn't prevent you from being arrested or detained. You don't even have to get all conspiracy-theory for this one; I'm not suggesting that the big bad government is going to go round up innocent dissenters and throw them in jail on purpose (that could conceivably happen down the line, but we don't need to go there for this argument). But wrongful convictions happen already. Like, all the time. Now granted, if the surveillance information were totally public, it could actually be used to prove innocence in some of those cases ("Need an alibi? Just look at my NSA phone records"), but that's not going to happen. The information will be tightly held (as it has been for the past 6-7 years) and will be available (often as circumstantial evidence) only to convict, not to acquit.
(3) Surveillance has a chilling effect on free expression, creativity, and spontaneity, even when the expression WOULD BE TOTALLY LEGAL. Incontrovertible evidence shows this to be the case, so even if you truly don't do anything illegal, provided you still want to express yourself (or--if you think you're immune to the chilling effects of surveillance--if you want to live in a more vibrant world where the people around you feel free to voice their opinions and express themselves without looking over their shoulders), the surveillance state is a bad idea.