The sword button is just the harvest/attack button, but for some items in the map you need specific tools for it to work.
