Did he perhaps already give you the hammer? You should be able to see it in the inventory as a weapon in that case. Else try to check your questlog to see what next step is needed, or try to go back to the secret garden to check if the next step now starts due to having the hammer.
