Just encountered this while building. I accidentally built a ladder, had them tear it down and replaced it with the pillar I originally wanted. However, the game continued to use it as a ladder. I couldn't build on top of it, and the characters would use the opening to go up and down the raft.