Found a workaround, If I let them first attack (that means I need them to allow to hit me once) then I am free to fight back, without everyone turning hostile.
