Pretty interesting idea! I think it could work very well as a fully-fletched game with more mechanics and more enemy variety. I do think the switches should have a more explicit outcome, because it favors trial and error as the game is currently.
I also loved the "juice" of the swap mechanic, I think it was well implemented!
Well done on submitting!