I think this concept is great! I enjoy the wonder around and find enemies instead of it being wave based. Would be cool to add a reason to go back to your camp.
I did feel like my scale dropped from shooting a bit too quickly. To the point where I'd get big, but the moment I start firing I'm small again. Took away from the needing to balance your size aspect.