The windup of the hammer strike is WAY longer, so it is far slower than it looks.
There are other drawbacks of the hammer, the handle doesn't damage them so you have to be more precise, and it doesn't block arrows and maybe swords too... I'm not sure.