You're not supposed to delete buildings. The buildings support and maintain the units. In the future unsupported units will slowly die off.
The intent is that units act as a drain on your economy, requiring you to expand in order to support more units.