cute idea, I think it should be more apparent which move direction is the currently active one, and some kind of feedback for if you attempt to move in the inactive direction. it was hard for me to notice the red flashing, especially on the background color you have, and the arrows are so small, and both direction's dice are exactly the same size. pressing a button to move in the "off" direction, and nothing at all happening, just made the game feel broken.
I made it through a few levels!