Ah, it doesn't show this very well but you can actually click again on a tower that you selected that round to deselect it for destruction before the round starts
