The fisher is below the map in a little house. Move down from the castle, you'll see it. It has no stars near the door.
