To destroy the armor you could only use hammer. That would make it better for the people who dislike it.
but what if you don't have a hammer when you meet them? I guess they could appear only after you get a hammer, but then people who like using only the sword would be forced to spend a point on the hammer :(
