So it seems that it's just in combat. I have a sickle equipped, enter combat, cast a spell and the sickles durability is reduced.
That's how it should work, casting damaging abilities reduce weapon durability just like a regular attack.
