The idea works quite well! It's very fun, and puts on a challenge for the player! I really enjoy the art as well!
Here is my game if you wanna play it, sorry for linking it in the comment but itch is bugged and isn't displaying it in the comment: https://itch.io/jam/gmtk-2020/rate/698697#post-1648262