But I only found 3 planks and was sent back to the main island. Damn. I need to replay now.
Once you do that you unlock the ability to get something that lets you get across again.