Its a very good idea for the game! At first we were brainstorming and we had the idea that maybe the player is the dice and controls a boardgame indirectly, but we could not think of a good way to make it fun.
I think this is the way to go!
The execution is not stellar, I never really had time to think about what I want to roll and execute it. Maybe it would be better if the dice drop could be triggered by the player after making a plan where he wants to move the pawn? Then its still a hurry to execute it but it also gives you a chance to plan ahead.