that is an interesting idea! I am a bit vary of adding too many special rules but if I would I‘d make this gradual. Something like:
If the city has no resources skip the optional natural growth phase and instead remove 1-3 citizen tokens as people start to leave.
Tho if you look at real world examples, there are often people that can‘t or won‘t leave. Usually people don’t simply abandon their home. So this would require more thought to get right.