I really liked the concept you had here. For some reason the game just performed very poorly on my machine. Possibly look into how to optimize game performance, the most basic and effect means is usually static batching, reducing material instances, baking lighting, LOD (level of detail). The performance while I played really hindered my experience. I did try the game off stream and it did help the performance a bit. Overall, great idea!
I streamed it here: