One thing you don't need to worry about is your public image. Sure, it's almost always useless to concern yourself with what people think (especially since they so rarely do). However, at the moment, you're perceived as something of an exemplar: one committed to doing the right thing, for the right reasons. If you have the feeling that your life would be easier if you didn't have to do this, you are correct; and that's a good sign. Keep taking the high road. Do the right thing instead of the popular thing. Maintain your policy of being friends with those who are unpopular or who are getting a raw deal. Your example of living as one who is true to your own values counts for more than you know. Carry on. Don't worry about whether you're making an impact. You are, where it counts the most—in the lives of individuals who respect you.
For more info, visit Planet Waves.