Don't worry, you can fight with any type of bandit as you wish when you complete the bandits camp quest
