Do you think veganism for things which are not harmed directly isn't a problem?

I.E. things which are taken naturally, rather than grown in farms? Just wondering the different perspectives.