This is a bug, sorry! If you test the die after closing and opening the app it should work properly.
The bug is that edited dice have several hundred faces (that don't get used), and loaded dice have the right number of faces. So any time you make or change a die, it will do this when testing (until the app reloads). I keep meaning to fix this issue but I've not been able to get round to it yet, sorry!