I had the same issue. Enemies couldn't destroy it, either. They'd run up and hit it once then run away. I fixed it by editing the save file by hand. :( First I gave the whole settlement to another tribe and tried to capture it back, but my units just hit it once and stopped while it streamed arrows at them. So I ran a find for "under" to find the hexes and just cleared out the structure completely and resettled.
I'm not sure if it was necessary, but I cleared out all structures except walls and roads in that region so that I could start fresh. I "marked" them by issuing demolish commands so that they would be under construction in the save file, for identification.