Did you get the Magic Mirror from the Cave? You need that item to get back to the Pirate Ship after the initial encounter.
