I agree, I think workers specifically need to work differently. I think I'd resolve it with a button that toggles between controlling only workers and only the rest, with both reverting to simple AI otherwise (mine last thing you touched for workers, fight anything near you for fighters). Having to manually mine resources this way works as a joke, but one that I think is achieved by telling the player how to operate the toggle button after one level of sillyness.