No, you're at a good level. It's more so that they are a 'tough' enemy, until you get elemental based weapons and/or spells. You can avoid them if you change the time to day by sleeping in camp.
