Do you have the axe equiped?
Oh, the sword doesn't attack?
The sword button is just the harvest/attack button, but for some items in the map you need specific tools for it to work.
