Wow. Very unique idea. That's a mechanic with lots of potential. You should really polish it up a bit more and make a post jam release, because this is awesome! Two things that could be improved:
- The difficulty is really up to 11 right from the start. This needs to be toned down a bit especially at the start of the game
- Add sound and music, obviously :D
Good entry, and again this has potential :)