How would you annoy them? using magic?
Viewing post in I just had a really good idea regarding spells.
Not sure, but I think it'd be funny if it were related to what the player wore. If a wizard hat was added in the future and you wore it, then maybe their tolerance for you attacking them (incidentally or directly) would go up before they retaliated, thinking that you were one of their own. Conversely, if you were wearing the Wawsp queen helmet, she'd call the wizards names when you got close to them, making them attack you on-sight.