You can always keep repairing your buildings and it should take quite a while for ravens to completely destroy them. Also it just needs several archers to properly defend the town.
