The concept is solid, I like the idea of building a world with the dice; maybe you could be building towards an end goal/level, and you have to strategize how to build the path (e.g. each dice value has different number of enemies; where you place each dice on the grid changes the enemy types, etc).
I'd try to take the idea further!