The main problem is that the buildings would be useless, because you can't break them. So no unit could get inside, you know?
How would you fix that?
That's were i think the gates would be fun, break down the gates or just go through them to get into the castle. just an idea