Basically because it amounts to shouting, "Fight me looser!"
At-least that's how I interpreted it I get orcs are generally like that in how they show friendship in fantasy settings, though it is quite clear that in your world, they do not behave like that. Well, the whole two orcs I've seen any dialogue from.
And I generally feel bad about fighting people who are just trying to live their lives without actively trying to make the lives of others difficult. As far as I'm concerned those servants are either getting paid or they volunteered for it.
I tend to take thing far more out of context than I have to. And sorry it took me so long to notice that you actually asked why I felt bad, I expected you to treat it like a joke.