Precisely, many, many shooting people do not condone the shooting of raptors; unfortunately, the extreme actions of some (I like to think the minority) cast a shadow over the whole.
I don't think this is any different from any other sport, where the negative actions of the few have a disproportionate effect on the public's view of the sport as a whole. This includes fell running or any other countryside activities, though these tend not to include the deaths of animals, protected or not.
As for environmental and ideological opposition to activities, I try to block out (hard to do) emotive arguments and "polls" and follow scientific evidence and balance which is the better or the lesser of evils.
I am far from convinced that a ban on grouse shooting would be to the overall benefit of wildlife and environment.
My view, but I'm not going to fall out about it.