Thanks for the feedback!
The inspiration for the Manufactory was the Cats' Keep token, in that it's the hub of your empire you expand out from and you have to be careful not to let it be destroyed as you can't get it back. I do think you are correct that they might be a bit too easy to stop right now since they start off a lot weaker than Cats.
In future versions I definitely would look into giving them a backup option for if their buildings get destroyed.