I think you've got some potential to this idea, but it needs a bit more work to make it good. Fortification doesn't seem to matter (sure the building survives, but it gets damaged and you spend an action repairing instead of on building so I don't see the difference). The different buildings don't seem to change anything.