It's a good idea to err on the side of not annoying players with random unexpected hits from enemies. Have you considered taking a page from fighting games and having different hit/hurt-boxes? Then you could fine-tune each without affecting the other.